Reading a large binary file (> available RAM)

See more:

C#3.5

I am using C# . NET 4.0 on Win 7 with a quod core and 16GB of RAM.

I have a very large binary file (say 24GB). I wish to break it up so I can read it.

I am having problems. Can anyone help? Thanks!

private static int nChunks = 24;
byte[] chunks;
    for (int i = 0; i < nChunks; i++)
    {
        chunks = null;
        chunks = chunker.getMyChunks(i, newFileName, nChunks);
        doStuff(chunks);
    }
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;

namespace myName
{
    public static class chunker
    {
               public static byte[] getMyChunks(int i, string filePath, int nChunks)
        {
            byte[] buffer;
            FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
            try
            {
                int length = (int)fileStream.Length;                // get total file length. This stays constant.

                long fLength = fileStream.Length / nChunks + 1;     // get the length of one of your chunks
                long readPositionFrom = fLength * i;
                long readPositionTo = fLength * (i+1);
                int lenToRead = Convert.ToInt32(readPositionTo - readPositionFrom);

                buffer = new byte[lenToRead];                               // Create buffer. This is what contains the data that is returned.
                int count;                                                  // Actual number of bytes that have been read
                int offset = Convert.ToInt32(readPositionFrom);             // Start reading from from this position in fileStream.

                // read until Read method returns 0 (end of the stream has been reached)
                // http://msdn.microsoft.com/en-us/library/system.io.filestream.read.aspx
                while ((count = fileStream.Read(buffer, offset, lenToRead - offset)) > 0)
                {
                    // first arg: buffer. the byte array being read into.
                    // second arg: offset. start populating the array at some element other than the first.
                    // third arg: (lenToRead-offset). The number of elements to read into the array

                    offset += count;  // sum is a buffer offset for next reading
                }
            }
            finally
            {
                fileStream.Close();
            }
            return buffer;
        }
    }
}

---

ahh yes good point.

problem is that I dont know what the problem is (else I would have fixed it).

To avoid further confusion: I am trying to read a 24GB by breaking it up.

I am able to read the first chunk (ie i=0) and get an answer, however as I dont know what the contents of the file are I cant sanity check it .

When the code goes into the second loop (i=1) it fails on the try statement.

It is clearly something to do with the way I am calling Read.

Any help gratefully recieved.

Thanks!