Click here to Skip to main content
13,045,657 members (53,135 online)
Rate this:
Please Sign up or sign in to vote.
Hello All,

Following is my scenario:
I will get xml file in my source folder and i have to pick them up and process create new xml file and place it in destination folder. I have a window service in my application in which i have included file system watcher concept for performing above task.

Now i have two threads - Main thread(window service) and worker thread(Queue). File system watcher is in main thread and the CREATED event is setup to process the files. For watcher object Notify filter=Filename;Filter=*.xml;include sub-directories=false;internal buffer=32768.

Now when processing file i just get filename from filesystem watcher of main thread and provide it to worker thread(it is a background thread). In this worker thread i use queue concept where i will queue up files then process them one by one, when queue count > 0 then return back to watching folder step processing of all files is done.I opted for this strategy, since i gone through several sites and came to know buffer overflow might occur causing files not recognized. (Note:Files come from an external system - have no idea if file transfer is by move or copy or ftp) But still on initial stages of testing when external system was involved there was no issues.

But unfortunately after 15 days or so i just included a validation logic as per client request none of the remaining code was touched but at sometimes files are staying in source folder without getting processed. if i check service its in running state. If i cut & paste the file in source folder the event is triggering and it gets processed.This doesn't occur in my development environment when i run the code and check it. But this occurs in my test environment where i deployed my application.

At this point i am stuck and i do not know where and how to check...

Because the folder faces only few files and they will only be around 10-15kb. i dont beleive the cause might be buffer overflow. And i do not know if external system has changed the way the file placed to my source folder.

Anyways i need all of the suggestions/workarounds/links for reference i can check and exhaust from fellow coders/codeproject members.

Please help me fix this issue. As its soon about to go live. Client is already complaining this problem in UAT phase in test environment.

Thanks in advance.

********Update 12-04-2013 10:42 pm*********

Got information from external system that the file transfer happens via FTP. There is no imporvements even if buffer size increased from 32kb to 48kb. Now this file picking is the only issue client is facing. All other required functionalities performed by application is cleared successfully.

Should i include renamed or changed event of filesystemwatcher? If you say yes then which event should i included and tell why it was suggested by you.
Posted 9-Apr-13 16:43pm
Updated 5-Jun-17 22:02pm
Rate this: bad
Please Sign up or sign in to vote.

Solution 1

This is not really a "solution", but something you should better be aware of; please see these CodeProject articles:
FileSystemWatcher - Pure Chaos (Part 1 of 2)[^],
FileSystemWatcher - Pure Chaos (Part 2 of 2)[^].

Rate this: bad
Please Sign up or sign in to vote.

Solution 4

I am sorry that I only saw this question today. I have a several years' experience with FileSystemWatcher (and with the Win32 API functions when using C++ before C#).

These are the main areas to look out for:

1) As you said, make sure the code that handles notifications is as quick as possible, so as not to potentially miss notifications. The best solution, again as you have indicated, is to add the path to a "queue" which is then processed on another thread. (You need to make sure your queue(s) are managed in a thread-safe way. Use the handy Synchronized feature).

2) Because you may get a notification about a file as soon as it is created, but it still may be being written to (you may even get several more "change" notifications) watch out that you don't actually try to process the file before it has finished being written.

4) Error handling: FileSystemWatcher can fall over for all sorts of reasons, which may depend on where the folder is that you are watching. Especially since you are using it in a service you will want to ensure that it will at least report the error to alert someone, and/or be able to recover from problems itself where possible.

3) Existing files: FileSystemWatcher will not tell you about files that exist when you create it, so when your service starts you most likely want to look for files that already exist in your watched folders, because your service may have been stopped before finishing the queue (if it's written properly this SHOULD be possible!) or files may have arrived while it was stopped.

In order to overcome and handle these issues and limitations, where I have used FileSystemWatcher I have ended up with no less than 3 (of my own) threads, 2 queues and a dictionary!
I'm afraid I can't post the code itself, so this is my description of it:

1) The main application thread is where the FileSystemWatcher is created, as well as the other 3 threads.
2) A thread which processes the queues.
3) A thread which does nothing but wait and frequently check (every minute in my case) that the FileSystemWatcher is still operating.
4) A temporary thread used at start-up (or re-initialisation) for adding existing files to the "changed" queue.

1) A queue of strings for "changed" (or created) file paths.
2) A queue of strings for "removed" file paths.

Dictionary with file path as the key, holding information (such as last modified time and file size) for each file.

Respond to the FileSystemWatcher events as follows:

Changed / Created: Add "FullPath" to the "changed" queue.
Deleted: Add "FullPath" to the "removed" queue.
Renamed: Add "OldFullPath" to "removed" queue, and "FullPath" to "changed" queue.
OnError: Log exception and ensure "EnableRaisingEvents" is set to false.

My second thread is then running in a loop, to do some work every configurable number of milliseconds, to process the queues as follows:
1) Go through the "removed" paths queue, for each path dequeue it and remove the entry for the path in the dictionary if it exists.
2) Go through the "changed" paths queue in a similar way, and add (or update) information about the file in the dictionary.
3) Go through each path in the dictionary:
a) Check if the last modified time stored in the information is greater than a configurable number of milliseconds. If it is, then
b) Check that the file size has not changed since the information was last checked (and update it). If it has not changed, then
c) Check that the file is not locked, by attempting to temporarily open it read-write. If this succeeds
d) Process the file (do whatever you need to do with it) and
e) Finally, add the file path to the "removed" queue.

My third thread is always initialised immediately after the FileSystemWatcher. It uses the same filtering options to go through the folder being watched adding any files it finds to the "changed" queue. This thread exits when it has finished.

My fourth and final thread runs for the lifetime of the service like the first two. It is another loop with a waiting time (e.g. 1 minute). Each time it enters the loop, it checks that the FileSystemWatcher is active (EnableRaisingEvents is true - hence why I ensure it is forced back to false in OnError). If this is false, it checks that the folder to watch exists. If the folder does not exist it will log a warning, and then go back round the loop, i.e. wait for a minute and then check again. If/when the folder does exist, it will call the initialisation method, which recreates the FileSystemWatcher and runs the third thread (to check for existing files), and then log an information message to say that the file system watcher was successfully reset. (If reset fails, an error is logged, and this thread will exit because there's no point continuing).

Well, that's a rather long winded description, but I hope it is helpful.

Ian A Davidson 23-Apr-13 14:25pm
Once again, someone down-voting without any comment as to why. That is most disappointing. If there's something wrong with my article, please say so.
H.Brydon 23-Apr-13 22:58pm
Good effort, good explanation...

+5 from me to partially compensate...
Sairam Kannan 24-Apr-13 5:25am
Thanks IAN for your response and tips. It will help as future reference for me as well as fellow programmers. I have handled most of the cases you have specified as i already had performed required analysis due to reason that i have seen many of the issues in existing application using FileSystemWatcher in my project. So there is nothing much that can be considered left out...apart from using several threads and polling thread. I opted for 2 threads alone since it was small application and as for polling thread it does not face heavy loads so it was manageable to handle stuffs with couple of thread alone.
Rate this: bad
Please Sign up or sign in to vote.

Solution 3

Hi All,
After couple of days i was able to find a solution by replicating the same scenario in my development enviroment just the way it was on my test environment. I requested the team member of external system to FTP sample file i post from my archive test files to my polling folder.
I have previously read in msdn where it states that files that are copied or moved may sometime only will be recognised as RENAMED event. Felt this might be applicable and so apart from CHANGED event, i also included RENAMED event to process files. This solved my issue and no files are getting clogged now in polling folder.
Replicating issue is one thing helped me. And another way to crack your issue while dealing with filesystemwatcher concepts in your coding is to use small project to test your case by monitoring the folder you watch for changes and note them down. So you can construct code accordingly.
Rate this: bad
Please Sign up or sign in to vote.

Solution 5

I had the same problem and reading your solution I realized why. In our code we where listening to CREATED event filtering on XML files as you. This event is always triggered, but since most SFTP software supports continuation on errors the files are created using a temp extension like .PART or .TMP or similar for then to be renamed when it's fully transferred. This is the reason your RENAMED event with filter on XML is the one hitting your code as the file isn't named XML until renamed.
Lasse S. Jensen

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Top Experts
Last 24hrsThis month

Advertise | Privacy | Mobile
Web02 | 2.8.170713.1 | Last Updated 6 Jun 2017
Copyright © CodeProject, 1999-2017
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100