|
I have made a custom filestream class that creates a copy of a file and uses the copy as the source to rewrite the file while making changes to values that match a specific criteria. Here is the basic setup:
public class MixedCompressionStreamRewriter : FileStream
It uses two modes: Parse mode reroutes the stream into a buffer to be analyzed and potentially rewritten before it gets sent to the output stream. When parse mode is false, it simply copies the base stream to the output stream. It also allows analysis of compressed data inside the stream because the source files will often have such data. Because of these use cases I had to override the Read method as such (commented so that one can trace where everything goes wrong in a debug session):
The issue occurs while executing the ReadUint32 call in the code below:
public static RecordHeaderBase FromFile(BinaryReader reader, uint internalIDMask = 0xFFFFFFFF)
{
ModRecordConsolidator consolidator = reader.BaseStream as ModRecordConsolidator;
if (consolidator != null)
consolidator.EnableParseMode = true;
RecordTypeInfo typeInfo = (RecordType)reader.ReadUInt32();
if (consolidator != null)
consolidator.EnableParseMode = false;
if (typeInfo == RecordType.TES4)
return new GameHeader(reader);
else if (typeInfo.isContainer)
return new GroupHeader(reader, typeInfo, internalIDMask);
else
return new RecordHeader(reader, typeInfo, internalIDMask);
}
This call leads to the execution of the overridden read method below which is where the actual problem occurs:
public override int Read(byte[] array, int offset, int count)
{
int result;
if (compressionHeaderStream == null)
{
result = base.Read(array, offset, count);
if (!parseMode)
{
long equivalentBufferLocation = MemoryBufferStart + MemoryBuffer.Length;
if (Position > equivalentBufferLocation)
{
long bufferCopyBytes = Position - equivalentBufferLocation;
if (bufferCopyBytes > int.MaxValue)
throw new OverflowException();
offset += (count - (int)bufferCopyBytes);
MemoryBuffer.Write(array, offset, (int)bufferCopyBytes);
}
if (MaxBufferSize != 0)
if (MemoryBuffer.Length >= MaxBufferSize)
FlushToOutput();
}
}
else if (MemoryBuffer == null)
result = base.Read(array, offset, count);
else
result = MemoryBuffer.Read(array, offset, count);
return result;
}
For some reason the *** starred line causes the file stream to read several thousand bytes rather than just four (which is confirmed by the value of count during debug). I have no idea why. During the base class read process, it accesses the following overridden methods:
public override bool CanRead { get { return true; } }
public override bool CanSeek { get { return true; } }
I have no idea why the base method (which is Microsoft provided) would do this.
EDIT: Even weirder: the base method returns that it only read 4 bytes. It might help to know that ModRecordConsolidator in the first code block inherits from MixedCompressionStreamRewriter (the class whose code is shown)
modified 6-Nov-20 22:35pm.
|
|
|
|
|
I think you've over-engineered and should have stuck with FileStream (and an adapter pattern). You've been caught up in the fog of indirection.
Inheritance has proven to be less useful than first thought; and requires a lot more thought.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
I am not really understanding a detail of your advice here. "Fog of indirection"? I chose the simplest solution I could think of for my needs as inventing a separate class would require me sending additional parameters through a lengthy call chain involving multiple methods COMBINED with the need to use just a regular filestream to perform the same function at other times. The code worked previously. It stopped working after I reinstalled windows, Visual Studio and made some changes to add additional functionality which left that part of the code untouched.
|
|
|
|
|
See. Still no clarity. "Another class" is a psychological limitation, not a real one.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
Is your point to help or to show how superior your knowledge of programming is? No it isn't a psychological limitation. It is a practical one. The file stream behind the reader is used as both an indexer to keep instance data separate in a static dictionary structure AND as a way to call required additional functionality on the stream. What you suggest would require me to change a significant amount of code to pass currently unnecessary data, making it more complex that it currently is since currently I can handle separate use cases with the same sections of code (that was the reason I used the current approach). Either way the solution is complicated so why not try and fix the problem first? Unless you are saying that the problem can't be fixed using an inherited FileStream because of its "complex" nature.
|
|
|
|
|
An "adapter" pattern lets you manipulate the filestream; instead, you want to "act" like a filestream. Since you didn't write filestream, you really have less control.
You also outlined 2 distinct functions. So, filestream in fact, is now schizophrenic.
Anyway, your problem seems to be you "request" "count" bytes, read, then don't bother to check "result" (actual bytes read) against "count"; and then proceed to calculate a new offset using "count" (instead of result).
Not "superior"; I'm just not the coddling kind. Have a nice day.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
I don't need to be coddled lol. I just need something beyond vague references as I did not major in CS and there are a few gaps in my learning (mostly self taught). And yes... I DID check "result" which was reported as 4 bytes. So that is not the issue.
The issue is that for some reason it is filling the buffer and counting it as being read when it isn't supposed to. I HAVE checked into that and confirmed it. I reported the issue on MSDN forums as something for them to look into and was told that inheriting/overloading FileStream itself can be unreliable if mixing in other unrelated functionality. I was advised to inherit Stream instead. This is similar to your original suggestion but serves my other needs as well.
|
|
|
|
|
We can't necessarily duplicate your problem from that code: we are lacking half the stuff that makes it work, such as the construction of your BinaryReader and it's FileStream.
So start by creating a minimum subset app which demonstrates the problem: uses the appropriate (cut-down) classes, dumps all the extraneous code, and shows the problem in as short a code as possible (but reflecting the structure you have in your existing code).
If that still shows the fault, show us the complete code so we can run it ourselves. If it doesn't, start looking at what you removed and put it back until it does ... then we can run it or it'll be fairly obvious what causes it.
Me? Without running it - and I can't - I'd guess that it's working but looks like it's failing for other reasons: if the return value is count then it's unlikely that the read is fetching considerably more data than that (somebody would have noticed by now). So you - and we - need a minimum app to show the problem so it can be looked at closely.
Sorry, but there is nothing I can do with those code fragments to prove your problem even exists!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Yes... it is even hard for ME to duplicate as it doesn't happen in all circumstances. I have tried my class on other files successfully. The one in this particular debug was the only one giving me the issue at the time and there is nothing I can isolate. See my other recent posts if you are curious but I did solve the issue with a workaround while I change the class to inherit from Stream instead of FileStream for the final solution.
|
|
|
|
|
By "several thousand", do you mean 4096?
The FileStream class buffers the file as it reads it. By default, it uses a 4Kb buffer.
FileStream buffers input and output for better performance.
You can pass a custom buffer size to the constructor if you want to override this behaviour. However, if you make the buffer too small, it could have a negative impact on performance.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Yes. Exactly. I noticed that when I was debugging it fills the buffer and moves the Position to match the location that it WOULD be at if I was reading to the end of the buffer even though it returns that it only read 4 bytes and should have only advanced the position 4 bytes. It seems to be some behavioral bug. I posted the issue on MSDN and they said that inheriting FileStream for any complex situation such as mine is a bad idea and may result in strange behavior... I guess it did. I am now inheriting Stream instead.
|
|
|
|
|
I have compiled successfully a C# console project. And when I run this app is stopping at the line:
System.IO.File.WriteAllLines(@destFile, elem);
where @destFile is
string destFile
and elem is
string[] elem = new string[...]
The error is:
System.UnauthorizedAccessException
HResult=0x80070005
Message=Access to the path 'e:\' is denied.
Source=mscorlib
StackTrace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
at System.IO.File.WriteAllLines(String path, String[] contents)
Of course, I have ran this app as administrator rights, with the same results. What should I understand from this error ? And how can I solve it ? My C# knowledge is thin ...
|
|
|
|
|
You cannot write a file to "E:\"; that is not a valid file name.
|
|
|
|
|
What do you expect @destFile to be?
You do not need an "@" in front of a variable name unless you are trying to use a keyword as a variable name - and if you are doing that, then you need to rethink your coding style. To call something public or string would require the "@" prefix, because they are both specific words that the compiler understands - but destFile is not, and does not need it.
int @public = 666;
Console.WriteLine(@public);
...
string destFile = "...";
Console.WriteLine(destFile);
And if you don't assign a value to destFile before you use it as an output location, you will get an error - it needs to contain a valid filename, in a folder that is not restricted for security reasons:
string destFile = "D:\Test Data\byList.csv";
File.WriteAllLines(destFile, elem);
So use the debugger to examine exactly what you have in destFile before you try to write to the file and check it's a fully valid filename!
If you don't know how to use it then a quick Google for "Visual Studio debugger" should give you the info you need.
Put a breakpoint on the first line in the function, and run your code through the debugger. Then look at your code, and at your data and work out what should happen manually. Then single step each line checking that what you expected to happen is exactly what did. When it isn't, that's when you have a problem, and you can back-track (or run it again and look more closely) to find out why.
Sorry, but we can't do that for you - time for you to learn a new (and very, very useful) skill: debugging!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Thank you for detailed explanation, the code is not written by me, so I try to adapt it as much as I can.
|
|
|
|
|
I don't know what happened to the reply you sent to me, so I reproduce it below:
Quote: Believe me I have tried a lot of combination (e.g. "D:\\tempx\\" or "d:\\tempx"), the same result.
It seems to me you are confused between directory names and file names. You cannot use Write of any sort to write into a directory, you must use a proper file name. So something like "E:\\mytextfile.txt", "D:\\tempx\\mytextfile.txt", is what is needed.
|
|
|
|
|
Yes, I figure out that and now I am trying to move on.
|
|
|
|
|
I have a website that I am trying to extract some data from and then run some things on it. They do not seem to have any API's to do this. So take this site: abc
if I view the source and look down to line 71 at this time I basically want this piece:
<script data-react-helmet="true" type="application/ld+json">
[{
"@context": "http://schema.org/",
"@type": "ItemList" ,
"itemlistElement":
[ [{
"@type": "ListItem",
"position" : 1,
"name" : "November 2020",
"item": [{
"@type": "Thing",
"name" : "General Hospital 11/03/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-11/03-general-hospital-110320"
},{
"@type": "Thing",
"name" : "General Hospital 11/02/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-11/02-general-hospital-110220"
}]
}],[{
"@type": "ListItem",
"position" : 2,
"name" : "October 2020",
"item": [{
"@type": "Thing",
"name" : "General Hospital 10/30/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/30-general-hospital-103020"
},{
"@type": "Thing",
"name" : "General Hospital 10/29/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/29-general-hospital-102920"
},{
"@type": "Thing",
"name" : "General Hospital 10/28/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/28-general-hospital-102820"
},{
"@type": "Thing",
"name" : "General Hospital 10/27/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/27-general-hospital-102720"
},{
"@type": "Thing",
"name" : "General Hospital 10/26/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/26-general-hospital-102620"
},{
"@type": "Thing",
"name" : "General Hospital 10/23/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/23-general-hospital-102320"
},{
"@type": "Thing",
"name" : "General Hospital 10/22/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/22-general-hospital-102220"
},{
"@type": "Thing",
"name" : "General Hospital 10/21/20",
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/21-general-hospital-102120"
}]
So in theory I could grab the whole source into a file like:
string htmlContent = new System.Net.WebClient().DownloadString("https://abc.com/shows/general-hospital");
But then how do I filter out just the block shown or just the JSON piece so I can work with it?
Thanks.
JR
modified 4-Nov-20 14:11pm.
|
|
|
|
|
Have a look at the HtmlAgilityPack: it makes scraping sites a whole load easier: Html Agility pack | Html Agility Pack[^]
For example, you can extract all the links from a page with one line of code:
foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))
{
...
}It's pretty easy to use, and very powerful when you get used to it.
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Thanks for the reply. I believe I actually tried just that but the problem is they aren't href's, they are JSON in a script so I believe that returned nothing unless I'm missing it. So like this from it:
"url" : "www.abc.com/shows/general-hospital/episode-guide/2020-10/26-general-hospital-102620"
There is no href in it so it doesn't retrieve it.
|
|
|
|
|
It looks to me as if that library parses the HTML into a Document Object Model that you can navigate and extract any component parts from.
Have a look at the documentation. I'm sure there's a way to extract JSON, you just have to find it. Griff was only giving an example of how useful the library is.
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Bear in mind that a large proportion of websites have T+Cs that explicitly stop you doing that.
I used to do a lot of small web-scraping jobs for a variety of clients; first task was always scour the site T+Cs to check it was permitted.
When I was doing it a lot there weren't any 3rd party tools available, so it's just a case of pulling out the parts of the string you want. In your case I'd search for Member 569739 wrote: type="application/ld+json"
(either using a simple ".indexOf()" call or using RegEx), check how many such script blocks there were, and continue likewise to pull out the bits I need. Pretty basic string manipulation.
In this case, though, since the whole of what you want is JSON, just load it up using Newtonsoft.JSON and then access the objects directly:
JObject myJsonObject = JObject.Parse(myJsonText);
|
|
|
|
|
The application/ld+json block is meant to be scraped by other sites - at the very least by search engines:
JSON-LD - JSON for Linking Data[^]
I think it's unlikely that the site would include that block if they didn't want anyone to be able to use it. 
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
"meant to be scraped" - yes.
By whom? Always check the T+Cs. 
|
|
|
|
|
Thanks folks. yeah it gets more complicated by the fact that there are multiple of those same blocks other than right now the one I want is the third one. That said, after hours of digging, I did find API's that return JSON for this so coding around that as obviously it's going to be a lot cleaner and more reliable. Haven't run the code yet but fingers crossed this works better. Not sure why ABC hides references to their API's like they do.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
