Click here to Skip to main content
Click here to Skip to main content
Go to top

A Faster File.Copy

, 23 May 2014
Rate this:
Please Sign up or sign in to vote.
Improving file transfer speed for large files

Introduction

I develop and maintain a Media Automation system for a satellite broadcaster. I often need to transfer files about 5GB and larger from one location to another. I used File.Copy or File.Move making the typical assumption that Microsoft .NET had already written highly efficient code for me. I was wrong! I typically achieved transfer speeds of about 60GB/hour over a Gigabit network connection depending on other traffic and disk activity. I thought I was doing reasonably well. I wasn't actually waiting for these files while they were going through various stages of processing so I didn't really care.

Then I had to do a mass conversion of a 50TB data store of 15,000 files. I thought 1000 hours! That would take 40 days! Well, it was a lot of data so I better get started. It was a two step process where I sent the file to a server where they were processed. An outside vendor provided a program which handled the processing and sent them back to the data store. I was only achieving 25GB/hour so I was beginning to get worried. Then I discovered they were only achieving 5GB/hour writing back. Well, I raised a stink quick. After a lot of wrangling, they changed some settings and all of a sudden they were achieving 160GB/hour!

Well, now I was flabbergasted. How did they achieve more than I was? I wiped the dust off of 45 years of software development and began a deeper consideration of system architecture and software processing. I did a quick search on CodeProject and MSDN without finding anything. The first thing that came to mind was using larger buffer sizes in file read and write to reduce overhead.

Wow! My first test was written in a few minutes and achieved 250GB/hour. Now that was more like it.

I need to explain the production environment so you have a better understanding of the throughput numbers. My numbers may not duplicate on your system. My network source for the files was a Harmonic MediaGrid, which is a high density parallel network storage system. It is designed to serve multiple servers with high speed access. The destination was a Linux based media playout server. This system is not in production yet so there was little contention for resources. I was working on a dual quad core XEON Windows 2008 Server with 16GB RAM. The program performance used an average 25% CPU of all eight cores with this routine. The gigabit network card used an average 555Mb/sec for both simultaneous send and receive.

After I did some enhancements to my program, I retested the File.Move throughput at a shockingly slow 25GB/hour. The 1M buffer size achieved 256GB/hour. A 512KB buffer was almost as fast. Any buffer size smaller than that has rapid throughput drop off.

I hope you enjoyed my little story and it brought a smile to your face. Here is the code:

/// <summary> Time the Move
/// </summary> 
/// <param name="source">Source file path</param> 
/// <param name="destination">Destination file path</param> 
public static void MoveTime (string source, string destination)
{
    DateTime start_time = DateTime.Now;
    FMove (source, destination);
    long size = new FileInfo (destination).Length;
    int milliseconds = 1 + (int) ((DateTime.Now - start_time).TotalMilliseconds);
    // size time in milliseconds per hour
    long tsize = size * 3600000 / milliseconds;
    tsize = tsize / (int) Math.Pow (2, 30);
    Console.WriteLine (tsize + "GB/hour");
}

/// <summary> Fast file move with big buffers
/// </summary>
/// <param name="source">Source file path</param> 
/// <param name="destination">Destination file path</param> 
static void FMove (string source, string destination)
{
    int array_length = (int) Math.Pow (2, 19);
    byte[] dataArray = new byte[array_length];
    using (FileStream fsread = new FileStream 
    (source, FileMode.Open, FileAccess.Read, FileShare.None, array_length))
    {
        using (BinaryReader bwread = new BinaryReader (fsread))
        {
            using (FileStream fswrite = new FileStream 
            (destination, FileMode.Create, FileAccess.Write, FileShare.None, array_length))
            {
                using (BinaryWriter bwwrite = new BinaryWriter (fswrite))
                {
                    for (; ; )
                    {
                        int read = bwread.Read (dataArray, 0, array_length);
                        if (0 == read)
                            break;
                        bwwrite.Write (dataArray, 0, read);
                    }
                }
            }
        }
    }
    File.Delete (source);
}

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Frank T. Clark
Systems Engineer Three Angels Broadcasting Network
United States United States
I am a Software Systems Design Engineer experienced with IEEE standards and the entire IEEE software development life cycle. Concept Exploration, Requirements, Design, Implementation, Test, Installation and Checkout, Operation and Maintenance. I enjoy working with people and solving customer problems.
 
I am currently employed as Information Technology Staff in the religious satellite broadcasting industry.

Comments and Discussions

 
QuestionSorry, I could not confirm the claim PinmvpSergey Alexandrovich Kryukov3-Jun-14 17:49 
AnswerRe: Sorry, I could not confirm the claim PinprofessionalFrank T. Clark4-Jun-14 2:29 
GeneralRe: Sorry, I could not confirm the claim PinmvpSergey Alexandrovich Kryukov4-Jun-14 5:33 
AnswerRe: Sorry, I could not confirm the claim PinprofessionalFrank T. Clark6-Jun-14 5:14 
GeneralRe: Sorry, I could not confirm the claim PinmvpSergey Alexandrovich Kryukov6-Jun-14 12:38 
QuestionFile.Move will be faster anyway PinmvpSergey Alexandrovich Kryukov3-Jun-14 15:23 
AnswerRe: File.Move will be faster anyway PinprofessionalFrank T. Clark4-Jun-14 2:24 
QuestionNot quite correct timing PinmvpSergey Alexandrovich Kryukov3-Jun-14 5:42 
AnswerRe: Not quite correct timing PinprofessionalFrank T. Clark3-Jun-14 6:29 
GeneralRe: Not quite correct timing PinmvpSergey Alexandrovich Kryukov3-Jun-14 7:05 
AnswerRe: Not quite correct timing PinprofessionalFrank T. Clark3-Jun-14 7:44 
GeneralRe: Not quite correct timing PinmvpSergey Alexandrovich Kryukov3-Jun-14 8:07 
AnswerRe: Not quite correct timing PinprofessionalFrank T. Clark3-Jun-14 8:41 
GeneralRe: Not quite correct timing PinmvpSergey Alexandrovich Kryukov3-Jun-14 11:12 
SuggestionUse "using", close will be called automatically Pinmemberalex2001_ts27-May-14 2:09 
GeneralRe: Use "using", close will be called automatically PinprofessionalFrank T. Clark27-May-14 10:28 
SuggestionRe: Use "using", close will be called automatically Pinmemberalex2001_ts27-May-14 10:53 
QuestionRe: Use "using", close will be called automatically PinprofessionalFrank T. Clark29-May-14 9:08 
AnswerRe: Use "using", close will be called automatically PinmvpSergey Alexandrovich Kryukov3-Jun-14 5:37 
AnswerRe: Use "using", close will be called automatically PinprofessionalFrank T. Clark3-Jun-14 6:18 
GeneralRe: Use "using", close will be called automatically PinmvpSergey Alexandrovich Kryukov3-Jun-14 7:05 
QuestionRe: Use "using", close will be called automatically PinprofessionalFrank T. Clark29-May-14 9:18 
AnswerRe: Use "using", close will be called automatically Pinmemberalex2001_ts29-May-14 10:36 
GeneralRe: Use "using", close will be called automatically PinprofessionalFrank T. Clark29-May-14 12:19 
QuestionWhy array_length * 2 Pinmemberwvd_vegt26-May-14 1:26 
AnswerRe: Why array_length * 2 PinprofessionalFrank T. Clark26-May-14 2:20 
AnswerRe: Why array_length * 2 PinprofessionalFrank T. Clark27-May-14 10:24 
GeneralThanks PinmemberShamanKann24-May-14 2:39 
QuestionMy Vote of 5 PinprofessionalManikandan1023-May-14 23:37 
QuestionSee BufferedStream Pinprofessionaljfriedman23-May-14 15:31 
AnswerRe: See BufferedStream PinprofessionalFrank T. Clark24-May-14 9:39 
GeneralMy vote of 5 PinmemberIvandro Ismael23-May-14 14:05 
GeneralRe: My vote of 5 PinprofessionalFrank T. Clark24-May-14 9:36 
GeneralMy vote of 5 PinprofessionalVolynsky Alex23-May-14 11:54 
GeneralRe: My vote of 5 PinprofessionalFrank T. Clark24-May-14 9:34 
GeneralRe: My vote of 5 PinprofessionalVolynsky Alex24-May-14 9:47 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140921.1 | Last Updated 23 May 2014
Article Copyright 2014 by Frank T. Clark
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid