Click here to Skip to main content
Sign Up to vote bad
good
See more: C#3.0
Hi
I am trying to merge two huge files each of size around 8 GB. But when it exceeds the RAM memory, either the system crashes or it throws an exception.
The background of the code is that it uses an array list to store multiple lines from a very huge notepad and then process the same in the memory. I also tried processing it directly in the file, but it took days to complete it. so I left that option.
Can you please suggest me any other ways to process this type of huge memory operation. Also is there any other ways to do these memory operation apart from using RAM, such as by using disk memory (not by using any File)?
 
Should I use any other object other than Arraylist so that I can use disk memory rather than physical memory?
 
Updating the question:
My question in general terms. How can I process a file size of 16 GB in background memory without using any text files to write and read the data in a computer which has a RAM of around 2 GB.
 
My Requirement:
It is for appending.
It goes like this...
1) For each line in FILE 1, pick certain data
2) Search for the selected data in FILE 2 entirely until the data is matched
If found, select the line which has the matched data and do some process and insert the processed data of FILE2 in the current line of the the FILE 1 at a particular position
 
Instead of editing FILE1 directly (because time consumption is extreamly very high), I want to read the data from both the file and store it in memory and then do the process for each line of the FILE 1 in memory by comparing it with FILE 2 and then write the entire data in the memory to a new output file.
 
I tried this process with small sized files and it worked very fast but failed when trying with huge files due to out of memory
 
Thanks in advance
Posted 18 Jan '12 - 19:42
Edited 30 Jan '12 - 19:44

Comments
Mikael Svenson - 30 Jan '12 - 2:44
What's your merging requirement? Sort, append?
NaveenSoftwares - 31 Jan '12 - 1:44
I have updated the question accordingly

2 solutions

To read large file you can use streamReader.
Just do reader.ReadToEnd() on a background thread, and display a marquee-type progress bar while processing.
 
or
 
//use Buffer with stringbuilder to read textfiles
int bufferSize = 1024;
var sb = new StringBuilder();
var buffer = new Char[bufferSize];
var length = 0L;
var totalRead = 0L;
var count = bufferSize;
using (var sr = new StreamReader(@"C:\Demofile.txt"))
 {
    length = sr.BaseStream.Length;
    while (count > 0)
    {
      count = sr.Read(buffer, 0, bufferSize);
      sb.Append(buffer, 0, count);
      totalRead += count;
    }
 } 
  Permalink  
Comments
NaveenSoftwares - 29 Jan '12 - 22:13
Thanks Prasad, But it still stores the data in buffer memory (RAM) and if it is reading 8 GB file, it will soon run out of memory and will throw an exception. My concern is that how can I process a file size of 16 GB in memory without using any text files in a computer which has a RAM of around 2 GB.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Your Filters
Interested
Ignored
     
0 Sergey Alexandrovich Kryukov 464
1 Mahesh Bailwal 373
2 Maciej Los 215
3 CPallini 175
4 Rohan Leuva 175
0 Sergey Alexandrovich Kryukov 9,402
1 OriginalGriff 7,204
2 CPallini 3,933
3 Rohan Leuva 3,211
4 Maciej Los 2,743


Advertise | Privacy | Mobile
Web03 | 2.6.130516.1 | Last Updated 31 Jan 2012
Copyright © CodeProject, 1999-2013
All Rights Reserved. Terms of Use
Layout: fixed | fluid