Click here to Skip to main content
15,741,818 members
Please Sign up or sign in to vote.
4.00/5 (2 votes)
See more:
Hi,

I load a file into memory and I have noticed that it can take radically different amounts of time.

I read whole file as a byte stream and file is 1G in size. Usually the first time I load a newly created file - it loads fast - in about 1 second.

Then when I rebuild my code and load it again - the same file can take 30 to 60 secs to load.

Can anyone tell me why?
Posted
Comments
Sergey Alexandrovich Kryukov 11-Mar-11 11:10am    
How any one can answer your question if you don't even explain what is "load", in what component, how you read. Where is your code? It's hardly a reading time, but rather rendering or something...
Let me guess, how many more questions will you need to provide adequate problem report? How about at once?
--SA
OriginalGriff 11-Mar-11 11:25am    
Hi S.A.! "how many more questions will you need" - has he been posting questions and deleting them when he has an answer? If I had known that, I might not have answered at all... I find that so rude!
Sergey Alexandrovich Kryukov 11-Mar-11 12:20pm    
Thank you for this note. I'll blacklist OP, but such behavior deserves a total ban.
However, how do you know that OP is deleting the post? It could be done by someone exactly (for example somebody irritated by OP's behavior)? I understand this should not be done if there is an Answer already.

Why are you still answering then? Just asking.
--SA
OriginalGriff 11-Mar-11 14:52pm    
No, no - I was responding to your comment!
I saw it after I had replied, checked the OP questions count, and assumed that you had seen similar questions by him before. We are both leaping to conclusions I think. Sorry if I misunderstood and have caused any confusion.
Sergey Alexandrovich Kryukov 11-Mar-11 22:11pm    
No, I did not want to leap to conclusion, I tried to warn against leaping to conclusion (I asked, how do you know who removed the post). I thought you're certain that OP behave incorrectly, so I could trust you and blacklist him. If you're not sure, I won't.
I'm sorry if I misunderstood, too.
--SA

1 solution

If it is a newly created file, then there is a good chance that the bulk of it is still in the cache: it is in your computer's memory as well as on disk (in fact it may not have reached the disk yet).

If it is in memory, then the PC does not need to read it from the hard drive, so it can load it a lot quicker.

The second time, you have rebuilt your program, so some of the memory taken up by the file has been re-used by VS. So when you access the file, it has to be read from the HDD.

Hard drives are very very much slower to access than main memory!
 
Share this answer
 
Comments
maninmoon 11-Mar-11 11:19am    
OriginalGriff,

Yes - thought it might be that BUT, when I run for third time, u would imagine file was still in memory as you suggest - however, that also takes long time...
OriginalGriff 11-Mar-11 11:23am    
Depends on what else is going on. Bear in mind that 1Gig is a lot of data. It is entirely possible that loading the file (and whatever else your program does with the data) causes the file to drop out of the cache...
maninmoon 11-Mar-11 11:42am    
var fileStream = new FileStream(filename,
FileMode.Open,
FileAccess.Read,
FileShare.Read,
16 * 1024,
FileOptions.SequentialScan);

byte* pBytes = (byte*)Form1.GlobalBuffer.ToPointer();
Form1.GlobalMS = new UnmanagedMemoryStream(pBytes, (long)length, (long)length, FileAccess.ReadWrite);
fileStream.CopyTo(Form1.GlobalMS);

It's this last line that takes the time.

I have 24G ram - so I don't think that's the issue....
Sergey Alexandrovich Kryukov 11-Mar-11 12:41pm    
I'll believe that if you time responses programmatically and report the results. I suspect this is not the pure reading time but something processed after. 1Gb is not much for disk read operation, but heavy in memory. What GlobalMS does?
--SA
maninmoon 11-Mar-11 12:46pm    
I do time it - and have pinpointed the .CopyTo line above. It is the unmanagedMemoryStream.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900