Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
See more: C#3.0
Hi
I am trying to merge two huge files each of size around 8 GB. But when it exceeds the RAM memory, either the system crashes or it throws an exception.
The background of the code is that it uses an array list to store multiple lines from a very huge notepad and then process the same in the memory. I also tried processing it directly in the file, but it took days to complete it. so I left that option.
Can you please suggest me any other ways to process this type of huge memory operation. Also is there any other ways to do these memory operation apart from using RAM, such as by using disk memory (not by using any File)?
 
Should I use any other object other than Arraylist so that I can use disk memory rather than physical memory?
 
Updating the question:
My question in general terms. How can I process a file size of 16 GB in background memory without using any text files to write and read the data in a computer which has a RAM of around 2 GB.
 
My Requirement:
It is for appending.
It goes like this...
1) For each line in FILE 1, pick certain data
2) Search for the selected data in FILE 2 entirely until the data is matched
If found, select the line which has the matched data and do some process and insert the processed data of FILE2 in the current line of the the FILE 1 at a particular position
 
Instead of editing FILE1 directly (because time consumption is extreamly very high), I want to read the data from both the file and store it in memory and then do the process for each line of the FILE 1 in memory by comparing it with FILE 2 and then write the entire data in the memory to a new output file.
 
I tried this process with small sized files and it worked very fast but failed when trying with huge files due to out of memory
 
Thanks in advance
Posted 18-Jan-12 20:42pm
Edited 30-Jan-12 20:44pm
v4
Comments
Mikael Svenson at 30-Jan-12 2:44am
   
What's your merging requirement? Sort, append?
NaveenSoftwares at 31-Jan-12 1:44am
   
I have updated the question accordingly
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 2

To read large file you can use streamReader.
Just do reader.ReadToEnd() on a background thread, and display a marquee-type progress bar while processing.
 
or
 
//use Buffer with stringbuilder to read textfiles
int bufferSize = 1024;
var sb = new StringBuilder();
var buffer = new Char[bufferSize];
var length = 0L;
var totalRead = 0L;
var count = bufferSize;
using (var sr = new StreamReader(@"C:\Demofile.txt"))
 {
    length = sr.BaseStream.Length;
    while (count > 0)
    {
      count = sr.Read(buffer, 0, bufferSize);
      sb.Append(buffer, 0, count);
      totalRead += count;
    }
 } 
  Permalink  
Comments
NaveenSoftwares at 29-Jan-12 22:13pm
   
Thanks Prasad, But it still stores the data in buffer memory (RAM) and if it is reading 8 GB file, it will soon run out of memory and will throw an exception.
 
My concern is that how can I process a file size of 16 GB in memory without using any text files in a computer which has a RAM of around 2 GB.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 DamithSL 421
1 OriginalGriff 220
2 Sergey Alexandrovich Kryukov 207
3 Peter Leow 185
4 Afzaal Ahmad Zeeshan 170
0 OriginalGriff 7,380
1 DamithSL 5,359
2 Sergey Alexandrovich Kryukov 4,944
3 Maciej Los 4,906
4 Kornfeld Eliyahu Peter 4,514


Advertise | Privacy | Mobile
Web03 | 2.8.141223.1 | Last Updated 31 Jan 2012
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100