Click here to Skip to main content
12,824,299 members (42,908 online)
Click here to Skip to main content
Add your own
alternative version

Stats

178.7K views
15.1K downloads
153 bookmarked
Posted 13 Aug 2009

A Faster Directory Enumerator

, 27 Aug 2009 CPOL
Rate this:
Please Sign up or sign in to vote.
Describes how to create a significantly faster enumerator for the attributes of all the files in a directory.

Introduction

The .NET Framework's Directory class includes methods for querying the list of files in a directory. For each file, you can also then query the attributes of the file, such as the size, creation date, etc. However, when querying files on a remote PC, this can be very inefficient; a potentially expensive network round-trip is needed to retrieve each file's attributes. This article describes a much more efficient implementation that is approximately 3x faster.

Background

Let's assume you are writing an application that needs to find the most recently modified file in a directory. To implement this, you might have a function similar to the following:

DateTime GetLastFileModifiedSlow(string dir)
{
    DateTime retval = DateTime.MinValue;
    
    string [] files = Directory.GetFiles(dir);
    for (int i=0; i<files.Length; i++)
    {
        DateTime lastWriteTime = File.GetLastWriteTime(files[i]);
        if (lastWriteTime > retval)
        {
            retval = lastWriteTime;
        }
    }
    
    return retval;
}

That function certainly works, but it suffers from some very poor performance characteristics:

  1. GetFiles must allocate a potentially very large array.
  2. GetFiles must wait for the entire directory's entries to be returned before returning.
  3. For each file, a potentially expensive query is sent to the file system. No attempt is made to perform any sort of batch query.

You might think that converting to DirectoryInfo.GetFileSystemInfos would improve item #3:

DateTime GetLastFileModifiedSlow2(string dir)
{
    DateTime retval = DateTime.MinValue;
    
    DirectoryInfo dirInfo = new DirectoryInfo(dir);

    FileInfo[] files = dirInfo.GetFiles();
    for (int i=0; i<files.Length; i++)
    {
        if (files[i].LastWriteTime > retval)
        {
            retval = lastWriteTime;
        }
    }
    
    return retval;
}

This doesn't change anything however: the objects returned by GetFiles() are not initialized with any data, and will all query the file system the first time any property is accessed.

Making it Faster

The attached test application includes the FastDirectoryEnumerator class in FastDirectoryEnumerator.cs. Using the GetFiles method, we can write the equivalent of our first slow method.

DateTime GetLastFileModifiedFast(string dir)
{
    DateTime retval = DateTime.MinValue;
    
    FileData [] files = FastDirectoryEnumerator.GetFiles(dir);
    for (int i=0; i<files.Length; i++)
    {
        if (files[i].LastWriteTime > retval)
        {
            retval = lastWriteTime;
        }
    }
    
    return retval;
}

The FileData object provides all the standard attributes for a file that the FileInfo class provides.

Making it Even Faster

Use one of the overloads of the EnumerateFiles method to enumerate over all the files in a directory. The enumeration returns a FileData object.

Below is an example of the same method using FastDirectoryEnumerator:

DateTime GetLastFileModifiedFast(string dir)
{
    DateTime retval = DateTime.MinValue;

    foreach (FileData f in FastDirectoryEnumerator.EnumerateFiles(dir))
    {
        if (f.LastWriteTime > retval)
        {
            retval = f.LastWriteTime;
        }
    }

    return retval;
}

Performance

The test application allows you to create a large number of files in a directory, then test the time it takes to enumerate using all three methods. I used a directory with 3000 files and ran each test three times to give the best answer possible for each test.

Using a path on my local hard drive resulted in the following times:

  • Directory.GetFiles method: ~225ms
  • DirectoryInfo.GetFiles method: ~230ms
  • FastDirectoryEnumerator.GetFiles method: ~33ms
  • FastDirectoryEnumerator.EnumerateFiles method: ~27ms

That is roughly a 8.5x increase in performance between the fastest and the slowest methods. The performance is even more pronounced when the files are on a UNC path. For this test, I used the same directory as the previous test. The only difference is that I referenced the directory by a UNC share name instead of the local path. At the time of the test, I was connected to my home wireless network.

  • Directory.GetFiles method: ~43,860ms
  • DirectoryInfo.GetFiles method: ~44,000ms
  • FastDirectoryEnumerator.GetFiles method: ~55ms
  • FastDirectoryEnumerator.EnumerateFiles method: ~53ms

That is roughly a 830x increase in performance, and more than 2 orders of magnitude! And, the gap only increases as the latency to the PC containing the files increases.

Why is it Faster?

As mentioned above, Directory.GetFiles and DirectoryInfo.GetFiles have a number of disadvantages. The most significant is that they throw away information and do not efficiently allow you to retrieve information about multiple files at the same time.

Internally, Directory.GetFiles is implemented as a wrapper over the Win32 FindFirstFile/FindNextFile functions. These functions all return information about each file that is enumerated that the GetFiles() method throws away when it returns the file names. They also retrieve information about multiple files with a single network message.

The FastDirectoryEnumerator keeps this information and returns it in the FileData class. This substantially reduces the number of network round-trips needed to accomplish the same task.

History

  • 8-13-2009: Initial version.
  • 8-14-2009: Added security checks, parameter checking, and the GetFiles method.
  • 8-24-2009: Fixed the AllDirectories search using GetFiles. Removed note about .NET 4.0 including something similar.
  • 9-08-2009: Fixed the AllDirectories search when filter is not * or *.*.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

wilsone8
Software Developer (Senior)
United States United States
I've been a software engineer since 1999. I tend to focus on C# and .NET technologies when possible.

You may also be interested in...

Comments and Discussions

 
GeneralRe: Great class, one annoying issue Pin
Niskov2-Jul-13 20:38
memberNiskov2-Jul-13 20:38 
SuggestionRe: Great class, one annoying issue Pin
wilsone83-Jul-13 4:52
memberwilsone83-Jul-13 4:52 
GeneralRe: Great class, one annoying issue Pin
Niskov3-Jul-13 5:08
memberNiskov3-Jul-13 5:08 
GeneralMy vote of 5 Pin
Need Guru Help14-Jun-13 5:25
memberNeed Guru Help14-Jun-13 5:25 
QuestionBest way to sort FileData[] by CreationDate Pin
JohnJayAjero30-Apr-13 11:09
memberJohnJayAjero30-Apr-13 11:09 
AnswerRe: Best way to sort FileData[] by CreationDate Pin
wilsone83-May-13 3:59
memberwilsone83-May-13 3:59 
GeneralMy vote of 5 Pin
Matt Watson (Stackify)15-Apr-13 18:24
memberMatt Watson (Stackify)15-Apr-13 18:24 
QuestionFile Enumerator Pin
Oscargt6-Feb-13 17:01
memberOscargt6-Feb-13 17:01 
I trying to get the folder (and subfolder size) instead of the (as in your example) the modified date. I actually have this code (newbie I know):

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
       Dim PDRFolder As New DirectoryInfo("C:\InputMedia")
       Dim filesInfo9() As FileInfo = PDRFolder.GetFiles("*.*", SearchOption.AllDirectories)
       Dim fileSizePDR As Long = 0
       Dim fileSize9 As Long = 0

       'start storage calculation for Playout PDR Folder

       For Each fileInfo9 As FileInfo In filesInfo9
           fileSize9 += fileInfo9.Length
       Next

       fileSizePDR = fileSize9

       lblBytes_PDR.Text = fileSizePDR.ToString
       lblSize_PDR.Text = FormatNumber(fileSizePDR / (1024 * 1024 * 1024), 2) + " GB" + "   (" + lblBytes_PDR.Text + " bytes)"

       lblFolder_PDR.Text = PDRFolder.ToString

       lblHours_PDR.Text = CStr(FormatNumber((fileSizePDR / (1024 * 1024 * 1024)) / 15, 2))

       'Convert value in hours and minutes for PDR Folder

       Dim value9 As Decimal = lblHours_PDR.Text
       Dim ts9 As TimeSpan = TimeSpan.FromHours(value9)
       Label5.Text = (String.Format("{0}h {1}m {2}.{3}s", ts9.Hours, ts9.Minutes, ts9.Seconds, ts9.Milliseconds))

       hor5 = fileSizePDR

       'End storage calculation for PDR Folder

       If Val(lblSize_PDR.Text) > 700 Then  'value in GB
           pbPDR.Visible = True          'show an alarm
           tbLog.Text &= System.DateTime.Now.ToString & vbCrLf
           tbLog.Text &= "LíThere is too much storage in PDR" & vbCrLf
           tbLog.SelectionStart = tbLog.TextLength
           tbLog.ScrollToCaret()
       Else
           pbPDR.Visible = False
       End If
   End Sub


But it lacks of performance on heavy folders. How can I improve it and make it faster? Your demo works on show the differences in speed even when it is only for the modified thing. What is the property for "file size" instead of the "writelastime"?
QuestionDepth first vs. Breadth first Pin
achanlon27-Nov-12 6:33
memberachanlon27-Nov-12 6:33 
GeneralMy vote of 5 Pin
desenvolvedor821-Nov-12 4:27
memberdesenvolvedor821-Nov-12 4:27 
GeneralMy Vote of 5 Pin
Juan R. Huertas10-Oct-12 2:32
memberJuan R. Huertas10-Oct-12 2:32 
GeneralMy vote of 5 Pin
pdoxtader17-Jul-12 5:21
memberpdoxtader17-Jul-12 5:21 
GeneralMy vote of 5 Pin
xyzabc1233215-Feb-12 16:31
memberxyzabc1233215-Feb-12 16:31 
QuestionCOM interop exceptions Pin
lwdaddio1-Feb-12 19:41
memberlwdaddio1-Feb-12 19:41 
QuestionPermissions Pin
Member 438704825-Oct-11 3:55
memberMember 438704825-Oct-11 3:55 
GeneralThank you for posting this code Pin
Member 776647422-Mar-11 3:54
memberMember 776647422-Mar-11 3:54 
GeneralThanks for posting... Pin
Ashi200230-Nov-10 1:54
memberAshi200230-Nov-10 1:54 
GeneralRe: Thanks for posting... Pin
ferrydebruin6-Apr-11 4:53
memberferrydebruin6-Apr-11 4:53 
GeneralMy vote of 5 Pin
Ashi200230-Nov-10 1:53
memberAshi200230-Nov-10 1:53 
GeneralDisposing resources [modified] Pin
Member 54073122-Oct-10 10:08
memberMember 54073122-Oct-10 10:08 
GeneralAccess to the path - denied Pin
danijel16-Oct-10 12:06
memberdanijel16-Oct-10 12:06 
GeneralRe: Access to the path - denied Pin
damage3119-Mar-11 14:58
memberdamage3119-Mar-11 14:58 
GeneralNice Work! Pin
Member 38053784-Aug-10 15:08
memberMember 38053784-Aug-10 15:08 
GeneralUnhandled Stackoverflow Exception on Windows 7 64bit Pin
Insomniac Geek4-May-10 12:47
memberInsomniac Geek4-May-10 12:47 
GeneralRe: Unhandled Stackoverflow Exception on Windows 7 64bit Pin
vdhaeyere28-Feb-11 5:52
membervdhaeyere28-Feb-11 5:52 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.170308.1 | Last Updated 27 Aug 2009
Article Copyright 2009 by wilsone8
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid