C# Discussion Boards - CodeProject

30-Jul-12 21:52

Hi
I’ve extremely log file that I need to parse, 700M
I need to read the file content and parse each line,
The problem is that I get exception System.outofmemory

In order to solved the memory issue, I think of reading the file in pages

I think of defile pageSize to be 100,000 lines
In order clean memory of array such as fileLinesContent, I think of defining it as null or to call Array.Clean(fileLinesContent)
Is it correct approach to clean memory of array?

string []fileLinesContent;
internal void ReadFile(int pageSize)
{

//To clean memory for garbage collector
fileLinesContent=null;
List<string> fileLines = new List<string>();
int linesNumRead=0;
string line;
while (linesNumRead < pageSize && !srLog.EndOfStream )
{

line = srLog.ReadLine();
linesNumRead ++;
}
fileLinesContent = fileLines.ToArray();
}
}

Eddy Vluggen30-Jul-12 23:40

30-Jul-12 23:40

Ronenb wrote:
Is it correct approach to clean memory of array?

Once the array is out of scope, the garbage-cleaner will reclaim that memory. Might not do it immediate, but when it thinks that it's required.

What are you going to do with the array of lines? Maybe there's a better solution, like passing a stream.

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

Ronenb30-Jul-12 23:45

30-Jul-12 23:45

hi the array (string []fileLinesContent) is a member of the class, not local member in func,

so my question is how to clean it in order free memory and read next lines in files

i'm reading the lines because i need to know when the line end and parse the element in each line

ronen

Eddy Vluggen31-Jul-12 0:10

31-Jul-12 0:10

Ronenb wrote:
the array (string []fileLinesContent) is a member of the class, not local member in func,

Then it's memory will be released once the object goes out of scope.

Ronenb wrote:
so my question is how to clean it in order free memory and read next lines in files

You can't dispose memory manually; there's a garbage-collector to do so. Very long strings are moved to the Large Object Heap, and simply stay there.

Ronenb wrote:
i'm reading the lines because i need to know when the line end and parse the element in each line

I'd recommend reading from a Stream, until you encounter that newline-character, and process it immediately. There's little need to keep all that information in memory.

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

Ronenb31-Jul-12 3:02

31-Jul-12 3:02

Thank u all for the replay
i think it is better to use the stream option as you suggested

can you send me example how to stream a file for each line?

should i use StreamFile, StreamRead or MemoryStream service?

thanks
ronen

Eddy Vluggen31-Jul-12 3:12

31-Jul-12 3:12

Ronenb wrote:
should i use StreamFile, StreamRead or MemoryStream service?

A FileStream[^] would be the most appropriate. Next, you use a StreamReader, which has a

<a href="http://msdn.microsoft.com/en-us/library/system.io.streamreader.readline.aspx">ReadLine</a>[<a href="http://msdn.microsoft.com/en-us/library/system.io.streamreader.readline.aspx" target="_blank" title="New Window">^</a>]

method.

Ronenb wrote:
can you send me example how to stream a file for each line?

See the links in this post, I don't provide copy/paste answers.

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

Pete O'Hanlon31-Jul-12 3:13

Pete O'Hanlon

31-Jul-12 3:13

Are you sure you don't deliberately sometimes supply copy/paste answers that are for the wrong question?

*pre-emptive celebratory nipple tassle jiggle* - Sean Ewington

"Mind bleach! Send me mind bleach!" - Nagy Vilmos

CodeStash - Online Snippet Management | My blog | MoXAML PowerToys | Mole 2010 - debugging made easier

Eddy Vluggen31-Jul-12 3:18

31-Jul-12 3:18

Failed to parse your question; can you slice it into simpler pieces for me?

To try and answer that; yes, I do sometimes post code that can be copied/pasted. Yes, I also supply the wrong answer sometimes.

Did I miss anything?

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

Pete O'Hanlon31-Jul-12 3:27

Pete O'Hanlon

31-Jul-12 3:27

Eddy Vluggen wrote:
Did I miss anything?

Yes.

I had suggested that depending on how stroppy the OP had been, you could deliberately supply the wrong piece of code. Not that I'm advocating this of course. Oh no. Definitely not. Noway. Nosirree.

*pre-emptive celebratory nipple tassle jiggle* - Sean Ewington

"Mind bleach! Send me mind bleach!" - Nagy Vilmos

CodeStash - Online Snippet Management | My blog | MoXAML PowerToys | Mole 2010 - debugging made easier

Eddy Vluggen31-Jul-12 5:05

31-Jul-12 5:05

Ronenb31-Jul-12 3:21

31-Jul-12 3:21

thanks

Ed Hill _5_31-Jul-12 3:20

Ed Hill _5_

31-Jul-12 3:20

this is a console app that may do what you are after, replace the console.writeline with a call to your parse method

class Program
{
    static void Main(string[] args)
    {
        string filePath = @"D:\ReadMe.txt";
        using (var streamReader = new System.IO.StreamReader(filePath))
        {
            while (!streamReader.EndOfStream)
            {
                var toParse = streamReader.ReadLine();
                System.Console.WriteLine(toParse);
            }
        }
        System.Console.ReadLine();
    }
}

Ronenb31-Jul-12 3:32

31-Jul-12 3:32

ok that this is Basically what i did
srLog is type of StreamReader
so my problem was that i didint need to read all file content to memory and then process it Smile | :)

SQL

I was thinking it is more efficient  to read all file content (with one stroke) and work on it then to access file system for each line(ReadLine) and process it
Isn’t more expensive?

while (!srLog.EndOfStream)
 {
     line = srLog.ReadLine();
             fileLines.Add(line);
 }
 fileLinesContent = fileLines.ToArray();

Ed Hill _5_31-Jul-12 3:40

Ed Hill _5_

31-Jul-12 3:40

Being honest i don't know which way would perform better, your origional post involved calling ReadLine many times in a loop any way and you were getting memory issues. This way should not be less efficient than the origional proposed solution.
I'd suggest you try it with something many times larger than your expected log file size, if the performance is good then don't stress too much about how you could make it better. If the file IO part of this App is not the bottle neck in the process then don't lose too much time optimising it.

Dave Kreskowiak31-Jul-12 4:29

Dave Kreskowiak

31-Jul-12 4:29

No, it's not. The way you were reading the file (line by line) and storing it in memory is no less expensive than processing that giant log one line at a time.

You're trying to read 700MB of data into memory and running into OutOfMemory problems. How efficient do you think that is?? By reading everything into memory all at once your solution only works on limited log sizes, dependant on system memory.

If you process every line, one at a time, without reading the entire file into memory, you can process log files of ANY size, up to the file size limit of the operating system and do it without requiring the machine to have terabytes of memory.

A guide to posting questions on CodeProject[^]

Dave Kreskowiak

Ronenb1-Aug-12 0:08

1-Aug-12 0:08

Got it

thanks all
Ronen

BobJanova31-Jul-12 0:31

BobJanova

31-Jul-12 0:31

700MB is not that large, you shouldn't run out of memory if you process it efficiently, even if you store the whole thing in memory. Calling List.ToArray is going to double the memory usage, though; you should either store it as a List all the time, or read it into an array to begin with (almost certainly the former).

However, I suspect you are doing some streaming task and you don't actually need the whole thing. Parse each line as it comes, and don't store it; instead store whatever information about a line you need to know, if anything.

Ennis Ray Lynch, Jr.31-Jul-12 4:52

Ennis Ray Lynch, Jr.

31-Jul-12 4:52

An array is a contiguous block of memory. Other than the fact that for large files all processing should be done in your while loop without retaining any of the file in memory your ToArray call could be killing you.

If you need Random Access to the File, look into Memory Mapped Files.

Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. I also do Android Programming as I find it a refreshing break from the MS.

"And they, since they Were not the one dead, turned to their affairs" -- Robert Frost

PIEBALDconsult31-Jul-12 6:09

PIEBALDconsult

31-Jul-12 6:09

I'll repeat that you probably shouldn't be keeping a bunch of rows in memory.

However, if you really need to, I suggest defining a class to hold an entry once it has been parsed. And don't use an array; use a List or a Queue, or if you're passing it to another class, consider using an event.