Click here to Skip to main content
15,916,600 members
Please Sign up or sign in to vote.
2.00/5 (1 vote)
See more:
I need to order according to the text of a line of text reading, when I read the file size of about 100MB in all normal, but when I read the file size is about 300MB when the memory error occurred.


function detail as below:
C#
var filePath = @"D:\junk\test.txt";
var fileSize = new FileInfo(filePath).Length;
var viewSize = fileSize
List<string> rowArray = new List<string>();
using (var mm = MemoryMappedFile.CreateFromFile(filePath, FileMode.Open, null, 0, MemoryMappedFileAccess.Read))
using (var stream = mm.CreateViewStream(0, viewSize, MemoryMappedFileAccess.Read))
using (var reader = new StreamReader(stream, Encoding.UTF8))
{
    rowArray.Add(reader.ReadLine());
}

The rowArray is provided to the LISTVIEW data source, LISTVIEW uses a virtual mode


What I have tried:

I use the third party software to read the file, whether it is the size of 100MB or 300MB size, everything is normal


Why I don't use this method of File.ReadAllLines is that it takes 28 seconds to read the 300MB file, and the Third party software takes about 10 seconds
So I used the MemoryMappedFile technology, but unfortunately, the 300MB file size out of memory error.
Posted
Updated 3-Jan-17 15:50pm
v6
Comments
OriginalGriff 3-Jan-17 4:51am    
Answer updated - comments are not currently being displayed, it;s a known bug and they are working on it.

Why are you making everything so complicated?
You create a new List of strings, to which you add a single string: the entire text content of the file, after memory mapping it and creating two streams form the mapped file.
Which is the equivalent of saying:
C#
List<string> rowArray = new List<string>();
rowArray.Add(File.ReadAllText(filePath));
Try that, and see if your memory problem goes away...

I need to read one line by one line and show it in ListView

That isn't what your code is doing, not even slightly.
And that very definitely isn't what you want to do at all, and is exactly why you are getting "out of memory" errors - never try to display that much information in one go. How many lines of text do you expect a file of 100MB (or worse 300MB) to contain? If the average is 80 characters per line - and it'll normally be a lot less - that's 1,250,000 lines for the 100MB file, and 3,750,000 for the bigger file. How do you expect your user to cope with that, never mind the system? Each of those lines is a separate control in a ListView! And that's why you run out of memory!

Don't do it. Page it, filter it, search it. But just throwing that much at a user and expecting them to cope it lazy and will only ever result in your user returning the software and demanding his money back...
 
Share this answer
 
v2
Comments
xuyunhai20160827 3-Jan-17 4:32am    
I need to read one line by one line and show it in ListView
Philippe Mori 3-Jan-17 9:36am    
The proposed code here will do exactly the same thing as your code but with less overhead.

By the way, why are you asking a question if you don't want to listen to experts...
xuyunhai20160827 3-Jan-17 21:32pm    
I tried your code just now,there's two questions.
1.The same computer, open a 300MB file size, takes about 28 seconds, while the third party software takes only about 10 seconds, I want to do with the performance of the same as the third party software.
2.If you use ReadAllText or ReadAllLines to read the data and load it into LISTVIEW, then I can't know the loading progress.
So I used the MemoryMappedFile technology, the fastest way of reading,but unfortunately, the 300MB file size out of memory error.
and now ,how to slove,thanks for your answer.

you can also read this arctics
Link as below:
http://stackoverflow.com/questions/4273699/how-to-read-a-large-1-gb-txt-file-in-net
The total memory that can be used by lists is limited to 2 GB. When using .NET 4.5 or later with 64-bit builds this limit can be increased with <gcAllowVeryLargeObjects> Element[^].

You may ask now why you run into this limit with files of about 300 MB size.

You have a string list. Strings are using UTF-16 encoding internally (two bytes per charater). If your input file contains only ASCII characters, the required memory to store the strings is twice the file size.

The list must also store the references to all strings. So you have 4 bytes (or 8 with 64-bit builds) per line.

The list will grow during reading. This requires re-allocation. Upon re-allocation, new memory is allocated, and the current content is copied before the old memory is released. During such operations you will probably get the out of memory exception. This can be avoided by setting the lists capacity which requires determination of the number of lines. This avoids re-allocation and allocating memory for more lines than actually required.

A solution for your problem might be using a list containing the line offsets for your memory mapped file. This list would only use 4 or 8 bytes per line. Because you are using a virtual list view, extract the strings in view by using the offsets (distance to next offset is line length).
 
Share this answer
 
Comments
OriginalGriff 3-Jan-17 4:52am    
It's worse than that - he's displaying each line separately in a ListView... :OMG:
Jochen Arndt 3-Jan-17 5:00am    
He uses a virtual list view.

Therefore, I suggested to use the memory mapped file with line offsets where the mapped file is the cache.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900