Click here to Skip to main content
Click here to Skip to main content

Automatically process incoming files

, 27 Oct 2007 CPOL
Rate this:
Please Sign up or sign in to vote.
Automatically process incoming files by using the DirectoryInfo class instead of FileSystemWatcher.

Introduction

Hello all. This is a little article on how to process incoming data automatically. In the short past, I've seen lots of people trying to use the FileSystemWatcher to trigger the data processing function in their software. However, many of them have to deal with the exception: "The process cannot access the file because it is being used by another process". We can fix this problem very easily. We only have to create a thread which reads the directory constantly, using the DirectoryInfo class. Next to that, we create a ProcessFile function, which processes the file and moves it into the 'processed' directory.

Using the code

Here we go:

private bool ProgramRunning = true;
private int FilesHandled = 1;

static void Main(string[] args)
{
    Program prog = new Program();
    prog.ReadDirectoryThread();
}

I use the field FilesHandled to count the number of files I've processed, just so that I can verify whether all files have been processed or not.

private void ReadDirectoryThread()
{
    while (this.ProgramRunning)
    {
        DirectoryInfo DirInfo = new DirectoryInfo(
            Path.GetDirectoryName(
            Process.GetCurrentProcess().MainModule.FileName)
            + "\\incoming\\");
        foreach (FileInfo FInfo in DirInfo.GetFiles("*.xml"))
        {
            Thread t = new Thread(new ParameterizedThreadStart(ProcessFile));
            t.Start(FInfo.FullName);
            while (t.ThreadState != System.Threading.ThreadState.Stopped)
            {
                //Thread.Sleep(5);
            }
        }
    }
}

We continuously check if new files are created. With the GetFiles() function, you can set a filter on a specific document type, for example, XML files. As soon as the directory contains documents, the ProcessFile function is called with the ParameterizedThreadStart() function. You can add a Thread.Sleep() in the t.ThreadState while loop to decrease the CPU usage; however, it is not recommended when you have to process a high amount of data files constantly.

void ProcessFile(object FileToProcess)
{
    string sFile = FileToProcess.ToString();
    bool isProcessed = false;
    // File processing
    try
    {
        // Process your file here.
    }
    catch
    {
        return;
    }
    // After the file is processed, we move it
    while (!isProcessed)
    {
        try
        {
            string ProcessedFilesDir = string.Format("{0}\\processed\\{1}_{2}.xml",
                Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName),
                DateTime.Now.ToString("dd_MM_yyyy_HH_mm_ss"), Guid.NewGuid().ToString());
            File.Move(sFile, ProcessedFilesDir);
            Console.WriteLine(FilesHandled + " - File " + 
                              sFile.ToString() + " processed\n");
            FilesHandled++;
        }
        catch
        {
            isProcessed = false;
        }
        finally
        {
            isProcessed = true;
        }
    }
}

In the first part in the code above, we can call the function to handle our data file, e.g., an import function or whatsoever. After the data has been handled successfully, we try to move the file to the 'processed' directory. In case the file has exclusive rights, the thread keeps trying to move the file until it's finally moved.

I've had a few cases where more than a hundred data files came in every second. Therefore, I added the DateTime.Now.ToString("dd_MM_yyyy_HH_mm_ss") and the Guid.NewGuid().ToString() functions, to create a proper filename for the processed file, without any risks on exceptions.

Because this is a console application, I used Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName). However, in Windows Forms, you can use Application.StartupPath instead.

History

  • 27 Oct 2007 - First version published.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Eric de Maar
Software Developer WDM
Netherlands Netherlands
Programmer from Groningen, the Netherlands.

Comments and Discussions

 
General100% CPU Usage! Pinmemberalien700018-Feb-09 7:48 
GeneralRe: 100% CPU Usage! Pinmemberalien700018-Feb-09 8:03 
GeneralHelfpul! Pinmemberleonvd2-Nov-07 21:11 
QuestionHow I have done this in the past. PinmemberBohicette230-Oct-07 11:57 
QuestionWhat is wrong with the FileMpnitor class PinmemberJoe Sonderegger28-Oct-07 22:18 
AnswerRe: What is wrong with the FileMpnitor class PinmemberEric de Maar28-Oct-07 23:31 
GeneralRe: What is wrong with the FileMpnitor class PinmemberJoe Sonderegger29-Oct-07 3:17 
GeneralRe: What is wrong with the FileMpnitor class PinmemberNico W29-Oct-07 4:16 
GeneralRe: What is wrong with the FileMpnitor class PinmemberJoe Sonderegger29-Oct-07 4:31 
GeneralRe: What is wrong with the FileMpnitor class PinmemberNico W29-Oct-07 4:45 
GeneralRe: What is wrong with the FileMpnitor class PinmemberJoe Sonderegger29-Oct-07 4:51 
GeneralRe: What is wrong with the FileMpnitor class PinmemberNico W29-Oct-07 5:10 
GeneralRe: What is wrong with the FileMpnitor class PinmemberJoe Sonderegger29-Oct-07 6:28 
GeneralRe: What is wrong with the FileMpnitor class PinmemberNico W29-Oct-07 6:47 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.141223.1 | Last Updated 27 Oct 2007
Article Copyright 2007 by Eric de Maar
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid