Writing to file in chunks

Question

0.00/5 (No votes)

See more:

Hi
I'm trying to write bytes to a file in chunks of chunkSize. This is really slow with large files, for example 9MB, and I need to find a way to make it much faster.

VB

For i As Integer = 0 To If(remainder = 0, 0, 1) + ((contents.Length - remainder) / blockSize)
    If contents.Length < i * blockSize Then
        IO.File.WriteAllBytes(outputPath, contents.Skip(i * blockSize).Take(blockSize).ToArray())
    Else
        IO.File.WriteAllBytes(outputPath, contents.Skip(i * blockSize).Take(contents.Length - (i * blockSize)).ToArray())
    End If
Next

The idea is to either write blockSize (4096) bytes, or whatever's left at the end if the number of bytes isn't a perfect multiple of blockSize.

What I have tried:

Various forms of this function

Posted 1-Jun-17 23:35pm

User 13204940

Updated 1-Jun-17 23:59pm

v2

Add a Solution

Comments

Richard MacCutchan 2-Jun-17 5:52am

What is contents, that you need to add those three methods to it each time?

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

OriginalGriff · Answer 1 · 2017-06-01T23:59:00

Why are you doing that?
All you are doing is slowing the whole thing down horribly - because you have to allocate a new array each time you write, and you destroy all previous data in the file: that;s what WriteAllBytes does!

Quote:
Creates a new file, writes the specified byte array to the file, and then closes the file. If the target file already exists, it is overwritten.

File.WriteAllBytes Method (String, Byte[]) (System.IO)[^]
Since your block size is 4096 bytes, that means you allocate over 2000 new arrays in order to do this!

First off, for .NET, 9MB is not that large a file: a single WriteAllBytes with a 9MB array will be as quick as the file system can cope with (and that's pretty fast, it knows a lot more about block sizes and caches than you do!).
Second if you want to write in chunks, then open a single stream outside the loop, and use FileStream.Write Method (Byte[], Int32, Int32) (System.IO)[^] to output each chunk - it provides offset and length parameters for just that purpose without needing to allocate memory for each chunk. But ... it'll still be slower than writing it all at once!