Click here to Skip to main content
15,390,470 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Visual studio doesn't allow you to append or split files larger than 2gb.

I'm trying to append a file using this method.

C#
FileStream fsSource = new FileStream(OriginalFilePath, FileMode.Append);
Byte[] bytePart = System.IO.File.ReadAllBytes(Part2FilePath);
fsSource.Write(bytePart, 0, bytePart.Length);


If the file Part2FilePath is more than 2gb i get the error:
"The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size."

I downloaded a program that can do it so I know it can be done. I just don't know if its possible with Visual Studio(C#). Any help would be appreciated. Thank.
Brian Cummings
Posted
Updated 4-Jan-11 3:11am
v2

Have you tried opening it with an appropriate Windows API function, instead of the built in File functions. These may not have the limit imposed (or may have a documented way around it). I suggest you have a look through the Windows API documentation, find a function that may work, and then look at this site to work out how use it:

http://pinvoke.net/[^]
   
Comments
brsecu 5-Jan-11 8:49am
   
Thank you for your help. I posted the solution i used below. I gave you a 5 out of 5 for your help.
Yes it can be done - I believe, I haven't tried it (I don't have many 2Gb files lying around, and those I do I don't particularly want to but together)

The problem is that you are trying to do it in a single chunk, via File.ReadAllBytes, which is (understandably) limited to 2Gb.

If you read it in chunks of say 100Mb then that will be a lot easier for the system to handle and should work fine.

See http://msdn.microsoft.com/en-us/library/system.io.filestream.read.aspx[^] and you'll be fine.
   
Comments
Manfred Rudolf Bihy 4-Jan-11 9:24am
   
That's what I suggested too! 5+
brsecu 5-Jan-11 8:49am
   
Thanks. I showed the solution below.
You have to read the file you want to append to the other in smaller chunks inside a loop. There is a limit of 2GB for one process (at least on 32bit machines). So if the function were to read the whole 2GB at once the process would run out of memory.

Besides that: fSource is a bit misleading here. It should be more like fDest since that file handle is the destination or the write operation. fSource should be the file Part2FilePath. Open that and read it in chunks of 100 - 500MB which you then append to fDest.

Best Regards,
Manfred
   
Comments
OriginalGriff 4-Jan-11 9:28am
   
"That's what I suggested too!"
Looks like I type faster than you do! ;)
5!
Is it for your personal use or is it for an application that you are developing? If it is for an in-house purpose, you might save time using System.Diagnostics.Process.Start() to call the tool that you downloaded. If the tool doesn't support command line, you could use some other free tools. 7-zip[^] is primarily a free compression software, but you can use it with a compression level of 0 to join files. Uncompressing will split the files.
   
Comments
brsecu 5-Jan-11 8:47am
   
Thank you for your help. I posted the solution i used below. I gave you a 5 out of 5 for your help.
Read this article. Hope it will benefit you
Split large XML files into small files[^]

Good Luck :)
   
Comments
brsecu 5-Jan-11 8:47am
   
Thank you for your help. I posted the solution i used below. I gave you a 5 out of 5 for your help.
Thank you everyone for your help. I always appreciate it and every answer here was a good one. I gave everyone a 5.

I am using an ftp api that downloads a 5g file everynight. The api requires on fail to make a new file then merge them. This is why i had that problem. I agree its not the best way but I have to use it per company policy.

I used OriginalGriff's answer. I ran a for loop broke it into 20 pieces and then put it back together with the original file.

Here is the code i used. I tried it on a short text file, zip file and exe. It worked on all three. I tried to format the code here I hope it looks ok. Thanks again.

Brian Cummings
for (int x = 0; x < 20; x++)
{
  try
  {
    using (FileStream fsSource = new FileStream(pathSource,
                        FileMode.Open, FileAccess.Read))
    {
    // Read the source file into a byte array.
    long fLength = fsSource.Length / 20 + 1;
    byte[] bytes = new byte[fLength];
    int numBytesToRead = (int)fLength;
    int numBytesRead = 0;
    while (numBytesToRead > 0)
    {
       fsSource.Position = readPosition;
       readPosition += fLength;
       int n = fsSource.Read(bytes, numBytesRead, numBytesToRead);
       
       // Break when the end of the file is reached.
       if (n == 0) break;
       
       numBytesRead += n;
       numBytesToRead -= n;
    }
    
   numBytesToRead = bytes.Length;
   // Write the byte array to the other FileStream.
   using (FileStream fsNew = new FileStream(pathNew + "00" +   x.ToString() + ".zip", FileMode.Create, FileAccess.Write))
   {
      fsNew.Write(bytes, 0, numBytesToRead);
      fsNew.Flush();
      fsNew.Close();
      fsNew.Dispose();
    }//using
 }//using
}//try
catch (Exception ex)
{
   MessageBox.Show(ex.Message);
   Console.WriteLine(ex.Message);
}

}//for loop x
   
v2

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900