Click here to Skip to main content
15,850,103 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Hi
I'm trying to write bytes to a file in chunks of chunkSize. This is really slow with large files, for example 9MB, and I need to find a way to make it much faster.

VB
For i As Integer = 0 To If(remainder = 0, 0, 1) + ((contents.Length - remainder) / blockSize)
    If contents.Length < i * blockSize Then
        IO.File.WriteAllBytes(outputPath, contents.Skip(i * blockSize).Take(blockSize).ToArray())
    Else
        IO.File.WriteAllBytes(outputPath, contents.Skip(i * blockSize).Take(contents.Length - (i * blockSize)).ToArray())
    End If
Next


The idea is to either write blockSize (4096) bytes, or whatever's left at the end if the number of bytes isn't a perfect multiple of blockSize.

What I have tried:

Various forms of this function
Posted
Updated 2-Jun-17 0:59am
v2
Comments
Richard MacCutchan 2-Jun-17 5:52am    
What is contents, that you need to add those three methods to it each time?

1 solution

Why are you doing that?
All you are doing is slowing the whole thing down horribly - because you have to allocate a new array each time you write, and you destroy all previous data in the file: that;s what WriteAllBytes does!
Quote:
Creates a new file, writes the specified byte array to the file, and then closes the file. If the target file already exists, it is overwritten.

File.WriteAllBytes Method (String, Byte[]) (System.IO)[^]
Since your block size is 4096 bytes, that means you allocate over 2000 new arrays in order to do this!

First off, for .NET, 9MB is not that large a file: a single WriteAllBytes with a 9MB array will be as quick as the file system can cope with (and that's pretty fast, it knows a lot more about block sizes and caches than you do!).
Second if you want to write in chunks, then open a single stream outside the loop, and use FileStream.Write Method (Byte[], Int32, Int32) (System.IO)[^] to output each chunk - it provides offset and length parameters for just that purpose without needing to allocate memory for each chunk. But ... it'll still be slower than writing it all at once!
 
Share this answer
 
Comments
[no name] 2-Jun-17 6:51am    
It's due to the encryption, I need the data to be in chunks of the same length as the key.
Richard Deeming 2-Jun-17 10:57am    
But you're overwriting the entire file each time through the loop!

And you're not changing the outputPath, so it's not like you're writing to different files.

If you just want the last block of bytes in the file, then replace the loop with its last iteration.

If you want all the bytes in one file, then just write them to the file. There's no difference between writing 9MB to a file in one hit, and writing it in multiple smaller chunks.

If you want the chunks in different files, then you're going to need to change the outputPath on each iteration.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900