Click here to Skip to main content
11,575,882 members (52,972 online)
Click here to Skip to main content

A Faster File.Copy

, 23 May 2014 CPOL 12.9K 34
Rate this:
Please Sign up or sign in to vote.
Improving file transfer speed for large files

Introduction

I develop and maintain a Media Automation system for a satellite broadcaster. I often need to transfer files about 5GB and larger from one location to another. I used File.Copy or File.Move making the typical assumption that Microsoft .NET had already written highly efficient code for me. I was wrong! I typically achieved transfer speeds of about 60GB/hour over a Gigabit network connection depending on other traffic and disk activity. I thought I was doing reasonably well. I wasn't actually waiting for these files while they were going through various stages of processing so I didn't really care.

Then I had to do a mass conversion of a 50TB data store of 15,000 files. I thought 1000 hours! That would take 40 days! Well, it was a lot of data so I better get started. It was a two step process where I sent the file to a server where they were processed. An outside vendor provided a program which handled the processing and sent them back to the data store. I was only achieving 25GB/hour so I was beginning to get worried. Then I discovered they were only achieving 5GB/hour writing back. Well, I raised a stink quick. After a lot of wrangling, they changed some settings and all of a sudden they were achieving 160GB/hour!

Well, now I was flabbergasted. How did they achieve more than I was? I wiped the dust off of 45 years of software development and began a deeper consideration of system architecture and software processing. I did a quick search on CodeProject and MSDN without finding anything. The first thing that came to mind was using larger buffer sizes in file read and write to reduce overhead.

Wow! My first test was written in a few minutes and achieved 250GB/hour. Now that was more like it.

I need to explain the production environment so you have a better understanding of the throughput numbers. My numbers may not duplicate on your system. My network source for the files was a Harmonic MediaGrid, which is a high density parallel network storage system. It is designed to serve multiple servers with high speed access. The destination was a Linux based media playout server. This system is not in production yet so there was little contention for resources. I was working on a dual quad core XEON Windows 2008 Server with 16GB RAM. The program performance used an average 25% CPU of all eight cores with this routine. The gigabit network card used an average 555Mb/sec for both simultaneous send and receive.

After I did some enhancements to my program, I retested the File.Move throughput at a shockingly slow 25GB/hour. The 1M buffer size achieved 256GB/hour. A 512KB buffer was almost as fast. Any buffer size smaller than that has rapid throughput drop off.

I hope you enjoyed my little story and it brought a smile to your face. Here is the code:

/// <summary> Time the Move
/// </summary> 
/// <param name="source">Source file path</param> 
/// <param name="destination">Destination file path</param> 
public static void MoveTime (string source, string destination)
{
    DateTime start_time = DateTime.Now;
    FMove (source, destination);
    long size = new FileInfo (destination).Length;
    int milliseconds = 1 + (int) ((DateTime.Now - start_time).TotalMilliseconds);
    // size time in milliseconds per hour
    long tsize = size * 3600000 / milliseconds;
    tsize = tsize / (int) Math.Pow (2, 30);
    Console.WriteLine (tsize + "GB/hour");
}

/// <summary> Fast file move with big buffers
/// </summary>
/// <param name="source">Source file path</param> 
/// <param name="destination">Destination file path</param> 
static void FMove (string source, string destination)
{
    int array_length = (int) Math.Pow (2, 19);
    byte[] dataArray = new byte[array_length];
    using (FileStream fsread = new FileStream 
    (source, FileMode.Open, FileAccess.Read, FileShare.None, array_length))
    {
        using (BinaryReader bwread = new BinaryReader (fsread))
        {
            using (FileStream fswrite = new FileStream 
            (destination, FileMode.Create, FileAccess.Write, FileShare.None, array_length))
            {
                using (BinaryWriter bwwrite = new BinaryWriter (fswrite))
                {
                    for (; ; )
                    {
                        int read = bwread.Read (dataArray, 0, array_length);
                        if (0 == read)
                            break;
                        bwwrite.Write (dataArray, 0, read);
                    }
                }
            }
        }
    }
    File.Delete (source);
}

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Frank T. Clark
Systems Engineer Three Angels Broadcasting Network
United States United States
I am a Software Systems Design Engineer experienced with IEEE standards and the entire IEEE software development life cycle. Concept Exploration, Requirements, Design, Implementation, Test, Installation and Checkout, Operation and Maintenance. I enjoy working with people and solving customer problems.

I am currently employed as Information Technology Staff in the religious satellite broadcasting industry.

You may also be interested in...

Comments and Discussions

 
QuestionSorry, I could not confirm the claim Pin
Sergey Alexandrovich Kryukov3-Jun-14 17:49
mvpSergey Alexandrovich Kryukov3-Jun-14 17:49 
AnswerRe: Sorry, I could not confirm the claim Pin
Frank T. Clark4-Jun-14 2:29
professionalFrank T. Clark4-Jun-14 2:29 
GeneralRe: Sorry, I could not confirm the claim Pin
Sergey Alexandrovich Kryukov4-Jun-14 5:33
mvpSergey Alexandrovich Kryukov4-Jun-14 5:33 
AnswerRe: Sorry, I could not confirm the claim Pin
Frank T. Clark6-Jun-14 5:14
professionalFrank T. Clark6-Jun-14 5:14 
GeneralRe: Sorry, I could not confirm the claim Pin
Sergey Alexandrovich Kryukov6-Jun-14 12:38
mvpSergey Alexandrovich Kryukov6-Jun-14 12:38 
QuestionFile.Move will be faster anyway Pin
Sergey Alexandrovich Kryukov3-Jun-14 15:23
mvpSergey Alexandrovich Kryukov3-Jun-14 15:23 
AnswerRe: File.Move will be faster anyway Pin
Frank T. Clark4-Jun-14 2:24
professionalFrank T. Clark4-Jun-14 2:24 
QuestionNot quite correct timing Pin
Sergey Alexandrovich Kryukov3-Jun-14 5:42
mvpSergey Alexandrovich Kryukov3-Jun-14 5:42 
AnswerRe: Not quite correct timing Pin
Frank T. Clark3-Jun-14 6:29
professionalFrank T. Clark3-Jun-14 6:29 
GeneralRe: Not quite correct timing Pin
Sergey Alexandrovich Kryukov3-Jun-14 7:05
mvpSergey Alexandrovich Kryukov3-Jun-14 7:05 
AnswerRe: Not quite correct timing Pin
Frank T. Clark3-Jun-14 7:44
professionalFrank T. Clark3-Jun-14 7:44 
GeneralRe: Not quite correct timing Pin
Sergey Alexandrovich Kryukov3-Jun-14 8:07
mvpSergey Alexandrovich Kryukov3-Jun-14 8:07 
AnswerRe: Not quite correct timing Pin
Frank T. Clark3-Jun-14 8:41
professionalFrank T. Clark3-Jun-14 8:41 
GeneralRe: Not quite correct timing Pin
Sergey Alexandrovich Kryukov3-Jun-14 11:12
mvpSergey Alexandrovich Kryukov3-Jun-14 11:12 
SuggestionUse "using", close will be called automatically Pin
alex2001_ts27-May-14 2:09
memberalex2001_ts27-May-14 2:09 
GeneralRe: Use "using", close will be called automatically Pin
Frank T. Clark27-May-14 10:28
professionalFrank T. Clark27-May-14 10:28 
SuggestionRe: Use "using", close will be called automatically Pin
alex2001_ts27-May-14 10:53
memberalex2001_ts27-May-14 10:53 
QuestionRe: Use "using", close will be called automatically Pin
Frank T. Clark29-May-14 9:08
professionalFrank T. Clark29-May-14 9:08 
AnswerRe: Use "using", close will be called automatically Pin
Sergey Alexandrovich Kryukov3-Jun-14 5:37
mvpSergey Alexandrovich Kryukov3-Jun-14 5:37 
AnswerRe: Use "using", close will be called automatically Pin
Frank T. Clark3-Jun-14 6:18
professionalFrank T. Clark3-Jun-14 6:18 
GeneralRe: Use "using", close will be called automatically Pin
Sergey Alexandrovich Kryukov3-Jun-14 7:05
mvpSergey Alexandrovich Kryukov3-Jun-14 7:05 
QuestionRe: Use "using", close will be called automatically Pin
Frank T. Clark29-May-14 9:18
professionalFrank T. Clark29-May-14 9:18 
AnswerRe: Use "using", close will be called automatically Pin
alex2001_ts29-May-14 10:36
memberalex2001_ts29-May-14 10:36 
GeneralRe: Use "using", close will be called automatically Pin
Frank T. Clark29-May-14 12:19
professionalFrank T. Clark29-May-14 12:19 
QuestionWhy array_length * 2 Pin
wvd_vegt26-May-14 1:26
memberwvd_vegt26-May-14 1:26 
AnswerRe: Why array_length * 2 Pin
Frank T. Clark26-May-14 2:20
professionalFrank T. Clark26-May-14 2:20 
AnswerRe: Why array_length * 2 Pin
Frank T. Clark27-May-14 10:24
professionalFrank T. Clark27-May-14 10:24 
GeneralThanks Pin
ShamanKann24-May-14 2:39
memberShamanKann24-May-14 2:39 
QuestionMy Vote of 5 Pin
Manikandan1023-May-14 23:37
professionalManikandan1023-May-14 23:37 
QuestionSee BufferedStream Pin
jfriedman23-May-14 15:31
professionaljfriedman23-May-14 15:31 
AnswerRe: See BufferedStream Pin
Frank T. Clark24-May-14 9:39
professionalFrank T. Clark24-May-14 9:39 
GeneralMy vote of 5 Pin
Ivandro Ismael23-May-14 14:05
memberIvandro Ismael23-May-14 14:05 
GeneralRe: My vote of 5 Pin
Frank T. Clark24-May-14 9:36
professionalFrank T. Clark24-May-14 9:36 
GeneralMy vote of 5 Pin
Volynsky Alex23-May-14 11:54
professionalVolynsky Alex23-May-14 11:54 
GeneralRe: My vote of 5 Pin
Frank T. Clark24-May-14 9:34
professionalFrank T. Clark24-May-14 9:34 
GeneralRe: My vote of 5 Pin
Volynsky Alex24-May-14 9:47
professionalVolynsky Alex24-May-14 9:47 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.150624.2 | Last Updated 23 May 2014
Article Copyright 2014 by Frank T. Clark
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid