Click here to Skip to main content
11,493,233 members (62,871 online)
Click here to Skip to main content

Automatically process incoming files

, 27 Oct 2007 CPOL 29.9K 169 18
Rate this:
Please Sign up or sign in to vote.
Automatically process incoming files by using the DirectoryInfo class instead of FileSystemWatcher.

Introduction

Hello all. This is a little article on how to process incoming data automatically. In the short past, I've seen lots of people trying to use the FileSystemWatcher to trigger the data processing function in their software. However, many of them have to deal with the exception: "The process cannot access the file because it is being used by another process". We can fix this problem very easily. We only have to create a thread which reads the directory constantly, using the DirectoryInfo class. Next to that, we create a ProcessFile function, which processes the file and moves it into the 'processed' directory.

Using the code

Here we go:

private bool ProgramRunning = true;
private int FilesHandled = 1;

static void Main(string[] args)
{
    Program prog = new Program();
    prog.ReadDirectoryThread();
}

I use the field FilesHandled to count the number of files I've processed, just so that I can verify whether all files have been processed or not.

private void ReadDirectoryThread()
{
    while (this.ProgramRunning)
    {
        DirectoryInfo DirInfo = new DirectoryInfo(
            Path.GetDirectoryName(
            Process.GetCurrentProcess().MainModule.FileName)
            + "\\incoming\\");
        foreach (FileInfo FInfo in DirInfo.GetFiles("*.xml"))
        {
            Thread t = new Thread(new ParameterizedThreadStart(ProcessFile));
            t.Start(FInfo.FullName);
            while (t.ThreadState != System.Threading.ThreadState.Stopped)
            {
                //Thread.Sleep(5);
            }
        }
    }
}

We continuously check if new files are created. With the GetFiles() function, you can set a filter on a specific document type, for example, XML files. As soon as the directory contains documents, the ProcessFile function is called with the ParameterizedThreadStart() function. You can add a Thread.Sleep() in the t.ThreadState while loop to decrease the CPU usage; however, it is not recommended when you have to process a high amount of data files constantly.

void ProcessFile(object FileToProcess)
{
    string sFile = FileToProcess.ToString();
    bool isProcessed = false;
    // File processing
    try
    {
        // Process your file here.
    }
    catch
    {
        return;
    }
    // After the file is processed, we move it
    while (!isProcessed)
    {
        try
        {
            string ProcessedFilesDir = string.Format("{0}\\processed\\{1}_{2}.xml",
                Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName),
                DateTime.Now.ToString("dd_MM_yyyy_HH_mm_ss"), Guid.NewGuid().ToString());
            File.Move(sFile, ProcessedFilesDir);
            Console.WriteLine(FilesHandled + " - File " + 
                              sFile.ToString() + " processed\n");
            FilesHandled++;
        }
        catch
        {
            isProcessed = false;
        }
        finally
        {
            isProcessed = true;
        }
    }
}

In the first part in the code above, we can call the function to handle our data file, e.g., an import function or whatsoever. After the data has been handled successfully, we try to move the file to the 'processed' directory. In case the file has exclusive rights, the thread keeps trying to move the file until it's finally moved.

I've had a few cases where more than a hundred data files came in every second. Therefore, I added the DateTime.Now.ToString("dd_MM_yyyy_HH_mm_ss") and the Guid.NewGuid().ToString() functions, to create a proper filename for the processed file, without any risks on exceptions.

Because this is a console application, I used Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName). However, in Windows Forms, you can use Application.StartupPath instead.

History

  • 27 Oct 2007 - First version published.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Eric de Maar
Software Developer WDM
Netherlands Netherlands
Programmer from Groningen, the Netherlands.

Comments and Discussions

 
General100% CPU Usage! Pin
alien700018-Feb-09 7:48
memberalien700018-Feb-09 7:48 
GeneralRe: 100% CPU Usage! Pin
alien700018-Feb-09 8:03
memberalien700018-Feb-09 8:03 
GeneralHelfpul! Pin
leonvd2-Nov-07 21:11
memberleonvd2-Nov-07 21:11 
QuestionHow I have done this in the past. Pin
Bohicette230-Oct-07 11:57
memberBohicette230-Oct-07 11:57 
QuestionWhat is wrong with the FileMpnitor class Pin
Joe Sonderegger28-Oct-07 22:18
memberJoe Sonderegger28-Oct-07 22:18 
AnswerRe: What is wrong with the FileMpnitor class Pin
Eric de Maar28-Oct-07 23:31
memberEric de Maar28-Oct-07 23:31 
GeneralRe: What is wrong with the FileMpnitor class Pin
Joe Sonderegger29-Oct-07 3:17
memberJoe Sonderegger29-Oct-07 3:17 
GeneralRe: What is wrong with the FileMpnitor class Pin
Nico W29-Oct-07 4:16
memberNico W29-Oct-07 4:16 
GeneralRe: What is wrong with the FileMpnitor class Pin
Joe Sonderegger29-Oct-07 4:31
memberJoe Sonderegger29-Oct-07 4:31 
GeneralRe: What is wrong with the FileMpnitor class Pin
Nico W29-Oct-07 4:45
memberNico W29-Oct-07 4:45 
GeneralRe: What is wrong with the FileMpnitor class Pin
Joe Sonderegger29-Oct-07 4:51
memberJoe Sonderegger29-Oct-07 4:51 
GeneralRe: What is wrong with the FileMpnitor class Pin
Nico W29-Oct-07 5:10
memberNico W29-Oct-07 5:10 
GeneralRe: What is wrong with the FileMpnitor class Pin
Joe Sonderegger29-Oct-07 6:28
memberJoe Sonderegger29-Oct-07 6:28 
GeneralRe: What is wrong with the FileMpnitor class Pin
Nico W29-Oct-07 6:47
memberNico W29-Oct-07 6:47 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.150520.1 | Last Updated 27 Oct 2007
Article Copyright 2007 by Eric de Maar
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid