Click here to Skip to main content
Click here to Skip to main content

CDirectoryChangeWatcher - ReadDirectoryChangesW all wrapped up

By , 11 May 2002
 

DirWatcher Sample App Screenshot

Introduction

This code wraps up the Win32 API function ReadDirectoryChangesW so that your application only has to worry about responding to the events that take place when a file or directory is added, removed, modified, or renamed.

This code will only work on Windows NT, Windows 2000, or Windows XP, and the directory you wish to watch must also reside on a WindowsNT, Windows 2000, or Windows XP computer.

The Classes

There are two classes that must be used together to watch a directory, they are: CDirectoryChangeWatcher and CDirectoryChangeHandler.

About CDirectoryChangeWatcher:

The class CDirectoryChangeWatcher does all the grunt work of watching a directory.  It creates a worker thread that is waiting for directory changes to take place via its use of the ReadDirectoryChangesW Win32 API function.   Multiple directories can be watched with a single instance of CDirectoryChangeWatcher, and directories can be added and removed from the watch at any time.  

When file change notifications are received, CDirectoryChangeWatcher 'dispatches' these notifications to your application via the the class CDirectoryChangeHandler(see below).

About CDirectoryChangeHandler:

The class CDirectoryChangeHandler receives notifications about file changes from CDirectoryChangeHandler.  You need to derive a class from CDirectoryChangeHandler in order for your application to handle the file change notifications.  
A single instance of a CDirectoryChangeHandler derived class can be used to handle notifications for multiple directory watches at the same time, or you may specify different CDirectoryChangeHandler derived classes for each watched directory.

The Interfaces

The following is the public interface of CDirectoryChangeWatcher:

class CDirectoryChangeWatcher{
public: 
   enum	{  //options for determining the behavior of the filter tests.
           FILTERS_DONT_USE_FILTERS     = 1, 
	   FILTERS_CHECK_FULL_PATH	= 2, 
	   FILTERS_CHECK_PARTIAL_PATH	= 4, 
	   FILTERS_CHECK_FILE_NAME_ONLY	= 8, 
	   FILTERS_TEST_HANDLER_FIRST	= 16,
	   FILTERS_DONT_USE_HANDLER_FILTER = 32, 
	   FILTERS_DEFAULT_BEHAVIOR	= (FILTERS_CHECK_FILE_NAME_ONLY ),
	   FILTERS_DONT_USE_ANY_FILTER_TESTS = (FILTERS_DONT_USE_FILTERS | 
                                                 FILTERS_DONT_USE_HANDLER_FILTER)
	};
  CDirectoryChangeWatcher(bool bAppHasGUI = true, 
                          DWORD dwFilterFlags = FILTERS_DEFAULT_BEHAVIOR);
  virtual ~CDirectoryChangeWatcher();//dtor
 
  DWORD	WatchDirectory( const CString & strDirToWatch, 
			DWORD dwChangesToWatchFor, 
			CDirectoryChangeHandler * pChangeHandler, 
			BOOL bWatchSubDirs = FALSE,
			LPCTSTR szIncludeFilter = NULL,
			LPCTSTR szExcludeFilter = NULL);

  BOOL IsWatchingDirectory(const CString & strDirName)const;
  int  NumWatchedDirectories()const; 

  BOOL UnwatchDirectory(const CString & strDirToStopWatching);
  BOOL UnwatchAllDirectories();

  DWORD	SetFilterFlags(DWORD dwFilterFlags);
  DWORD GetFilterFlags() const;

...
};


The class CDirectoryChangeHandler has the following interface:

class CDirectoryChangeHandler{
   ...
   BOOL UnwatchDirectory(); 
   ...
 protected:
  //override these functions:	
   //Notification related:
   virtual void On_FileAdded(const CString & strFileName);
   virtual void On_FileRemoved(const CString & strFileName);
   virtual void On_FileModified(const CString & strFileName);
   virtual void On_FileNameChanged(const CString & strOldFileName, 
                                   const CString & strNewFileName);
   virtual void On_ReadDirectoryChangesError(DWORD dwError);
   virtual void On_WatchStarted(DWORD dwError, const CString & strDirectoryName);
   virtual void On_WatchStopped(const CString & strDirectoryName);
   //Filter related:
   virtual bool On_FilterNotification(DWORD dwNotifyAction,
                                      LPCTSTR szFileName, LPCTSTR szNewFileName);
   ...
  };

To handle the events that happen when a file or directory is added, deleted, modified, or renamed, create a class derived from CDirectoryChangeHandler that does all of the things that you want to do when these events happen.

The Notifications

There are 7 notifications that you can receive when using these classes.

Notification
CDirectoryChangeHandler function
Notification Description
Flag(s) required to receive notification--
(dwChangesToWatchFor parameter
to 
CDirectoryChangeWatcher::<br>
      WatchDirectory()
)
Watch Started
On_WatchStarted
A directory watch has been started because  CDirectoryChangeWatcher::WatchDirectory was called.
N/A
Watch Stopped
On_WatchStopped
A directory watch has been stopped because 
 
      CDirectoryChangeWatcher::<br>
      UnwatchDirectory
or 
<br>
      CDirectoryChangeWatcher::<br>
      UnwatchAllDirectories
was called.

Can also be called when CDirectoryChangeHandler's desctructor is called and there are 1 or more directories being watched at that time.
** See the comments in DirectoryChanges.h near the On_WatchStopped function to avoid RTFM errors which can occur under certain circumstances.**
N/A
Directory Watch Error
On_ReadDirectoryChangesError
An error occurred while watching a directory. In this condition, the watch is stopped automatically.  You will not receive the On_WatchStopped notification.
N/A
File Added
On_FileAdded
A file was added to a watched directory (newly created, or copied into that directory).
FILE_NOTIFY_CHANGE_FILE_NAME and/or FILE_NOTIFY_CHANGE_DIR_NAME (for directories)
File Removed
On_FileRemoved
A file was deleted, or removed from the watched directory(ie: sent to the recycle bin, moved to another directory, or deleted)
FILE_NOTIFY_CHANGE_FILE_NAME  FILE_NOTIFY_CHANGE_DIR_NAME
File Name Changed
On_FileNameChanged
A file's name has been changed in the watched directory.  The parameters to this notification are the old file name, and the new file name.
FILE_NOTIFY_CHANGE_FILE_NAME  FILE_NOTIFY_CHANGE_DIR_NAME
File Modified
On_FileModified
A file was modified in some manner in a watched directory.  Things that can cause you to receive this notification include changes to a file's last accessed, last modified, or created timestamps. Other changes, such as a change to a file's attributes, size, or security descriptor can also trigger this notification.
FILE_NOTIFY_CHANGE_ATTRIBUTES
FILE_NOTIFY_CHANGE_SIZE
FILE_NOTIFY_CHANGE_LAST_WRITE
FILE_NOTIFY_CHANGE_LAST_ACCESS
FILE_NOTIFY_CHANGE_CREATION
FILE_NOTIFY_CHANGE_SECURITY

The Filters

One of the new features to this class is the ability to filter out notifications so that you can receive notifications only for files that you want to receive notifications about.  This feature enables you receive file notifications only for files you wish to know about based on the name of the changed file, or based on a function that you implement.  

Without a filter, you will receive notifications for any and all file changes, as specified by the combination of flags passed as the dwChangesToWatchFor parameter to the CDirectoryChangeWatcher::WatchDirectory function(which is the default by the way).

There are 3 types of filters: An Include Filter, an Exclude Filter, and a programmer implemented virtual function which can decide whether or not the appropriate CDirectoryChangeHandler::On_Filexxxxx() function is called.

The Include, Exclude filters:

The Include and Exclude filters are string parameters that are passed to CDirectoryChangeWatcher::WatchDirectory when a directory watch is started.  The filter is a pattern which can contain the DOS wildcard characters * and ?.   Multiple patterns can be specified by separating each pattern with a semi-colon (;).

The following are examples of valid pattern strings that can be used to filter notifications:

    "*.txt"   <-- specifies only a single file pattern.
    "*.txt;*.tmp;myfile???.txt;MySubFolder???\*.doc"  <-- specifies multiple file patterns.

Note that the supporting code for these string filters uses the PathMatchSpec Win32 API function to determine a pattern match. PathMatchSpec is implemented in shlwapi.dll version 4.71 or later. If you are running on NT4.0, and Internet Explorer 4.0 is not installed, a modified version of wildcmp is used instead.

What does the Include Filter mean?

The Include filter is used to tell CDirectoryChangeWatcher that you wish to receive notifications ONLY for files that match a certain pattern. All other file notifications are implicitly excluded because the file name does not match the 'Include Filter' pattern. ie: If you pass an 'Include Filter' of "*.txt", you will only receive notifications(File Added, File Removed, etc) for files with names that end with ".txt".  You will not be notified of changes to any other files.

Note: It's better to specify a NULL or empty string, than to specify an Include Filter of "*.*".

What does the Exclude Filter mean?

With the Exclude Filter, you can choose to tell CDirectoryChangeWatcher to explicitly ignore notifications for changes to files based on the name of the changed file.  For example, you can choose to ignore changes to .log files in a watched directory. To do so, you would specify an Exclude Filter of "*.log"

The Include and Exclude Filters are a powerful way of customizing the notifications that you wish to receive.  To test your pattern strings, there is a dialog provided as part of the sample application.  Press the "Test Filter Patterns..." button on the Sample App.

The Programmer Defined Filter:

You can also override the function : virtual bool CDirectoryChangeHandler::On_FilterNotification() . If On_FilterNotification returns true(the default), you will receive the notification.  If On_FilterNotification returns false, the file notification is ignored.  See the comments in the source code for more information about this function.

Filter Options

There are several options related to how CDirectoryChangeWatcher works with filters.  

Filter Flag
Flag Description
FILTERS_DONT_USE_FILTERS
Specifies that the string filters for the Include and Exclude filter are not checked.  All notifications are sent unless  CDirectoryChangeHandler::On_FilterNotification() returns false.
FILTERS_DONT_USE_HANDLER_FILTER
Specifies that CDirectoryChangeHandler::On_FilterNotification() is not tested. If the File name passes the test against the Include and Exclude filter, the notification is passed on.
FILTERS_DONT_USE_ANY_FILTER_TESTS
Specifies that NO filter tests are to be performed.  The Include and Exclude filters are not tested, and CDirectoryChangeHandler::On_FilterNotification() is not called.  This is a combination of FILTERS_DONT_USE_FILTERS and FILTERS_DONT_USE_HANDLER_FILTER flags.
FILTERS_NO_WATCHSTART_NOTIFICATION CDirectoryChangeHandler::On_WatchStarted won't be called
FILTERS_NO_WATCHSTOP_NOTIFICATION CDirectoryChangeHandler::On_WatchStopped won't be called
FILTERS_NO_STARTSTOP_NOTIFICATION CDirectoryChangeHandler::On_WatchStarted and CDirectoryChangeHandler::On_WatchStopped won't be called. Is a combination of FILTERS_NO_WATCHSTART_NOTIFICATION and FILTERS_NO_WATCHSTOP_NOTIFICATION.
FILTERS_TEST_HANDLER_FIRST
Specifies that CDirectoryChangeHandler::On_FilterNotification() is to be called BEFORE the file name is tested against the Include and Exclude filter patterns.  The default is that CDirectoryChangeHandler::On_FilterNotification() is tested AFTER only if the file name matches the patterns used that may have been specified as Include or Exclude Filters.
FILTERS_CHECK_FULL_PATH
Specifies that the entire file name and path is to be tested against the Include and Exclude Filter pattern.
FILTERS_CHECK_PARTIAL_PATH
Specifies that only a portion of the file path and name are to be tested against the Include and Exclude Filter.  The portion of the path checked is the part of the path following the watched folder name.  

For example, if you are watching "C:\Temp" (and are also watching subfolders) and a file named "C:\Temp\SomeFolder\SomeFileName.txt" is changed, the portion of the file name that is checked against the Include and Exclude filters is "SomeFolder\SomeFileName.txt"
FILTERS_CHECK_FILE_NAME_ONLY
Specifies that only the file name part of the file path is to be checked against the Include and Exclude filter.
FILTERS_DEFAULT_BEHAVIOR
Specifies the 'default' filter handling behaviour.
Currently, it's only set to FILTERS_CHECK_FILE_NAME_ONLY.

This implies that the Include/Exclude Filters are tested before On_FilterNotification, and that On_WatchStarted and On_WatchStopped are called.

To specify these options, see the constructor of CDirectoryChangeWatcher , or use the function CDirectoryChangeWatcher::SetFilterFlags() .

Note that the Filter Flags are used for the next call to CDirectoryChangeWatcher::WatchDirectory and that calling CDirectoryChangeWatcher::SetFilterFlags() will have no effect on any currently watched directories.

The sample app includes a dialog which allows you to test this out:

Thread Safety, and Message Pumps.

While CDirectoryChangeWatcher uses a worker thread to get the job done, all notifications are called in the context of the main thread . The 'main' thread is the thread that first calls CDirectoryChangeWatcher::WatchDirectory . CDirectoryChangeWatcher uses a hidden notification window to dispatch notifications from the worker thread to the main thread. Because it uses a window, the calling application must have a message pump implemented somewhere in the program's 'main' thread.

Console Applications

For console applications, or applications/threads without a message pump, CDirectoryChangeWatcher can still be used. Just pass false as the value of the bAppHasGUI parameter to the constructor of CDirectoryChangeWatcher . Instead of using a hidden notification window, CDirectoryChangeWatcher uses an additional worker thread. Note that when you pass false as the value of bAppHasGUI parameter to the constructor of CDirectoryChangeWatcher , that all CDirectoryChangeHandler functions are called within the context of a worker thread, and NOT the main thread.

CDirectoryChangeWatcher watcher(false); //safe to use in a console app.
				//
				//Note: CDirectoryChangeHandler functions are called 
				// 	in a worker thread.

A Sample:

class CMyDirectoryChangeHandler : public CDirectoryChangeHandler
{
public:
      	CMyDirectoryChangeHandler(){} 
      	virtual ~CMyDirectoryChangeHandler(){}

protected:
       	void On_FileNameChanged(const CString & strOldFileName, const CString & strNewFileName)
       	{
             MessageBox(NULL, _T("The file ") + strOldFileName + _T(" was renamed to: ") + 
                              strNewFileName,_T("Test"),MB_OK);
       	}
       	bool On_FilterNotification(DWORD dwNotifyAction, LPCTSTR szFileName, LPCTSTR szNewFileName)
       	{ 
	   //
            // This programmer defined filter will only cause notifications
            // that a file name was changed to be sent.
            //
		if( dwNotifyAction == FILE_ACTION_RENAMED_OLD_NAME ) 
                   return true;
              	return false;
	}
};

....
        CDirectoryChangeWatcher watcher;
        CMyDirectoryChangeHandler MyChangeHandler;
        watcher.WatchDirectory(_T("C:\\Temp"), 
                                 FILE_CHANGE_NOTIFY_FILE_NAME,
                                 &MyChangeHandler,
                                 FALSE, //<-- watch sub directories? 
                                 NULL, //<-- Include Filter
                    		 NULL);//<-- Exclude Filter

Fin

CDirectoryChangeWatcher was based on the FWatch example program in the SDK and uses an I/O completion port so that there will only be one worker thread (per instance of CDirectoryChangeWatcher ) for any number of watched directories.

When using this source code in your application, you only need to use CDirectoryChangeWatcher and class(es) derived from CDirectoryChangeHandler.  Any other classes in these source files are used to implement CDirectoryChangeWatcher.

Download the sample or source code for more details.

Acknowledgements

  • CDirectoryChangeWatcher is based on the FWatch sample in the SDK.
  • The sample application uses CFolderDialog by Armen Hakobyan.
  • CDirectoryChangeWatcher uses a modified version of wildcmp by Jack Handy

Feel free to email me with bugs, bug fixes, tips, comments, accolades, or admonitions at wesj@hotmail.com.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Wes Jones
Web Developer
United States United States
Member
No Biography provided

Sign Up to vote   Poor Excellent
Add a reason or comment to your vote: x
Votes of 3 or less require a comment

Comments and Discussions

 
You must Sign In to use this message board.
Search this forum  
    Spacing  Noise  Layout  Per page   
GeneralRe: Ask questions and share experiencememberYanay Sun14 Sep '11 - 1:07 
Thanks a lot!
I've tried again. The title of my email is "Asking for help from Yanay".
Hope to hearing from you soon!
QuestionSomething wrongmember_stefanu_6 Jul '11 - 10:26 
First of all, great work. But... there is a problem, and I quite can't put my finger on it because the code is too complicated Wink | ;) Still working on it.
 
All goes fine if you get events for one or few file(s) at a time in your watched folder; but if you are watching a folder with A LOT of activity (even worse, with subfolders, on a SSD drive), you miss events.
 
First, the PostThreadMessage fails miserably because the thread queue is limited to 10k events; since one gets 3 or more events per file when, say, it is created, this makes it happen for less than 3000 real file events.
 
So, besides writing a custom queue, one may wait for the notified thread to deal with some messages, then try and post again. This only adds up to the time spent in "On_FileAdded". So far, so good, but this is where you start missing events.
 
You can simulate this by adding a sleep call in DirectoryChanges.cpp, line 1906, just before the break. The more you 'sleep', the more events you will loose, eventually all of them.
 
As I said, I'm still working on understanding the problem, but can anyone help ?
 
Thanks,
Stefan.
AnswerRe: Something wrong [modified]memberWes Jones6 Jul '11 - 16:11 
Hey Stefan,
 
Wow, that's a lot of file activity! I guess I can feel good that it took this kind of activity before it shows a problem.
There are two windows of time a notification can be missed. The time inbetween ReadDirectoryChangesW returning, and the next call to ReadDirectoryChangesW being made. By doing a PostThreadMessage on the results from ReadDirectoryChangesW, I am able to keep this window of time to an absolute minimum. I guess the other window of time is when the thread already has 10k items posted against it & you post another one(that's the one that's missed).
 
To be honest, I've never heard of that limit before, and I'll take your word for it.
 
This is the kind of activity and limit I didn't have in mind... the thought being that handlers should be so fast, there shouldn't be time for 10k events to pile up in a post message queue.
 
Immediate thoughts on ways to tackle this are:
 
1) Evaluate the code in a profiler. What are your event handlers doing that take so long that a limit of 10k post message queue items are piling up behind it? The handlers need to be as fast as possible. Where are the bottlenecks?
 
2) Instead of 1 instance of DirectoryChangeWatcher watching all those folders/subfolders, think of ways to allow multiple instances of DirectoryChangeWatcher handle different sections of the folder. 1 instance of DirectoryChangeWatcher has 1 thread, and the 10k limit you're talking about it. Other instances of the watcher class will reduce the risk of that limit being reached.
Think about intances based on file folders, or instances based on the change flags your watching for, ie: Added changes-> watcher1, Removed Changes->watcher2, Modified->watcher3
Something like that.
 
3) Evaluate the consequences of changing the buffer size used w/ ReadDirectoryChangesW. I think it's 4096. It that buffer is full, it will take a certain # of cpu cycles to process that buffer and post each file change notification to the worker thread. This is the time when a change could be missed as ReadDirectoryChangesW won't get called until it finished w/ the buffer. A smaller buffer means it'll process it faster and call RDCW sooner. Of course, there's still a buffer filling up in the background somewhere and it could just be pushing the problem somewhere else. Record some statistics on how long it takes between calls to RDCW & evaluate if that's a big enough window for notifications to be missed.
 
4) The last approach is to use the Decorator Pattern on your DirectoryChangeHandler. This handler whould implement it's own thread pool and could itself use PostThreadMessage to it's own pool of threads so that 1 individual thread wouldn't ever have a chance to reach such a limit.
 
5) A mix of all of the above
 
Cheers,
Wes
GeneralRe: Something wrongmember_stefanu_6 Jul '11 - 21:49 
Thanks for the reply, and the advices, but unfortunately, I think none applies to my case.
 
Long story short : the event handler is copying the file to a different location, over the network, and how long it takes, it's out of my control. This is how I got to find out about the 10k limit.
 
Next, I tried to keep posting the message until the message queue accepts it, but then I started missing files because CDelayedNotificationThread::PostNotification takes too long; I'm not sure if files were missed previously too (I mean all calls to the On_FileAdded, not really processing them).
 
Bottomline, I think files are always missed, but maybe in smaller amounts (in happy cases, none), depending on how fast they are created and how long On_FileAdded takes; my real-life situation is not so extreme, but you can easily push it to an extreme where more than 90% of the files are missed.
 
Here's a quick test you can do right away with your sample, assuming your drive is not speed-lightning (this will cause the events to be generated over a longer period of time); put a breakpoint in the case branch FILE_ACTION_ADDED of ProcessChangeNotifications. Then copy and paste at least two or three small files to the watched folder. When the first file is created, the breakpoint fires. Wait a couple of seconds, then continue execution. You will never get the events for the rest of the files, and I may be very wrong, but I think the problem lies somewhere in the asynchronous call to RDCW.
 
I have created another test, using RDCW synchronously, just to see if it returns with lpBytesReturned=0, which it does at some point (this is why I think the problem lies in the async call to RDCW). I ended up increasing my buffer to 4M, so that it does not miss events; but this is not very safe either, since at some point it will always overflow.
 
The path I am going to take now is use RDCW synchronously, with a very large static buffer, and use a system pipe to handle the events, hopefully avoiding the 10k limit of the message queue.
 
I will get back with more info as soon as I have it.
GeneralRe: Something wrongmemberWes Jones7 Jul '11 - 5:43 
Hey Stefan,
 
The way to reproduce this that you describe sounds like you're setting a breakpoint in between calls to RDCW, which is when file notifications will be missed.
A problem w/ the On_FileAdded notification is that a file can be large enough that you receive a new file notification and windows is still writing chunks of that file to disk when you're handling the notification. That could increase the amount of time it takes for the file copy fo finish. To see that, copy & paste a 2GB file into a watched folder.
 
I would recommend that you create a class who's job it is to copy files in it's own background thread or threads. Your On_FileAdded handler will just post to that class. This way your On_FileAdded handler is super fast and you don't miss the notifications, and you can roll your own solution to regarding the limits your class can handle.
I did put a lot into designing around this very problem & that's my suggested solution in this case.
GeneralRe: Something wrongmember_stefanu_7 Jul '11 - 5:58 
Thanks. Indeed, I am setting the breakpoint or adding a sleep(10) to simulate an On_FileAdded call that takes longer. However, no matter how fast you handle the On_FileAdded, my guess is that there will be some events missed occasionally.
 
For now I rewrote the RDCW handling in my app without your classes, since I need a pretty targeted behavior, using several threads, pipes and synchronous calls.
 
Your classes are great and I have to thank you for the insight of how RDCW works, but I have been unable to use them; my guess is that when you first wrote your article, hardware and OS behavior were a bit different.
 
Thanks again for the article and your help.
GeneralRe: Something wrongmemberWes Jones7 Jul '11 - 7:05 
I'm glad it could help! Yeah, it has been a while since I wrote this, and who knows how things have changed. It might be time for a refactoring at a minimum, or to handle some things differently.
 
I am curious.. can you post any concrete numbers on what it takes for problems to occur? I doubt I'd be able to reproduce this scenario in on my home PC. What's the type of application and the kind of environment it's running in?
-Wes
GeneralRe: Something wrongmember_stefanu_7 Jul '11 - 22:01 
Sure. In my original scenario (which actually is a real-life situation), I had a folder with about 1000 subfolders, and files being created almost randomly in those folders. Since I cannot watch 1000 folders or more one by one, I needed a recursive watch on the root folder so that when files get created/modified/deleted, the same happens on a remote machine; sort of a replication mechanism.
 
First test : less than 50% (don't recall the exact numbers, but I think it was something like 20k out of 44k) of the files ended up on the remote machine.
 
So I began testing and stressing the app to find out why and what I could do about it. I then devised a simpler test, which is far more stressful than the real-life situation, just to see how bad it is. The second step was the breakpoint test I described earlier, but that even more exaggerated.
 
I had a folder with 6000 small files (actually one of the source files Wink | ;) ), named 00001.cpp to 06000.cpp, so I can trace them. I modified the sample app to OutputDebugString the contents of the ListBox as it is created, so I can paste it here, and made a few minor changes so it would compile nicely on VS2008, but no structural changes whatsoever; just stuff like 'for (int i=0; ...', then using i outside the loop; there were just a couple of them.
 
All events were checked in the dialog, and then, in the watched folder, I copy-pasted using the Windows Explorer my test folder with the 6000 files.
 
I tried on two machines; since I believe system performance plays an important role, here are the details :
 
1. Core2Duo 2 Ghz, 4GB RAM, SSD drive with good performance on small files, the copy-paste takes less than 10 seconds
 
2. Core i5 2.30 GHz, 8GB RAM, regular HDD drive; the copy-paste takes less than 20 seconds
 
Putting a 'Sleep(100)' in the handlers, just to simulate a somehow exaggerated lag, here's what I see on machine (1); the same happened on machine (2), but it was worse (meaning, even less events fired).
 
Here's an excerpt of the trace; before this point, everything looks ok.
 
[...]
File Added: C:\DirWatch\Test\1 - Copy (2)
File Added: C:\DirWatch\Test\1 - Copy (2)\00004.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)\00004.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)\02480.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)
File Added: C:\DirWatch\Test\1 - Copy (2)\02480.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)\02480.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)\02481.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)
[...]
 
As you can see, no events are fired for files between 4 and 2480.
 
And the last few lines of the trace are :
 
[...]
File Added: C:\DirWatch\Test\1 - Copy (2)
File Added: C:\DirWatch\Test\1 - Copy (2)\02495.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)\02495.cpp
File Added: C:\DirWatch\Test\1 - Copy (2)\02496.cpp
The thread 'Win32 Thread' (0xbf4) has exited with code 0 (0x0).
 
No sign of the rest of the files up to 6000. So you can see, this is a significant loss of events, not just accidental stuff; I'd say it's really close to 100%, unfortunately. This is obviously due to the lag and the small RDCW buffer; the only problem is that it look pretty hard to devise a proper mechanism.
 
On the other hand, I have advanced a little more on my implementation, with sync calls (to avoid buffer issues) and pipes (to avoid the 10k event limitation), and I am beginning to see the light (actually, it just became a 100W light bulb from the small candle it was yesterday) : no file loss in a test that ran overnight.
 
If I get the chance of finding some spare time, I will try to wrap things up in a demo app.
GeneralRe: Something wrongmemberWes Jones8 Jul '11 - 4:52 
Ah, the ol' somebody copy & pasted 6000 files routine. Gets 'em every time! I'm surprised by the number of notifications that appear to be missed.
I think it's time for an update to the code. I'll use your this info & this kind of test as a benchmark & perhaps will email you re: any nuggets of wisdom you gained when you solved the problem. Hopefully it's not a limitation of the async thing I'm doing w/ the completion ports as my whole implementation is based on that.
GeneralRe: Something wrong [modified]member_stefanu_8 Jul '11 - 5:30 
The only thing I can suggest is use pipes instead of message queues; seems to be working fine for me, and a bigger buffer.
 
Also, in my case, a stopping mechanism was not necessary, so the RDCW thread keeps on looping, hence the synchronous call approach.
 
What I've been doing can be summed up like this (in some sort of pseudo-code Blush | :O )
WorkerThread
   do
      read event from pipe
      process event
   while TRUE
end
 
WatcherThread
   create communication pipe
   create worker thread
   do
      RDCW synchronous call
      dispatch events through the pipe to the worker thread
   while TRUE
end
Not sure if this is sane enough, but it runs in a service, and the stop service command has no problem in killing the threads, even if they're in a wait state, so I won't drill into that anymore for now.
 
And one more thing... the RDCW buffer is 16M; used it as static variable in the tests, but will go to GlobalAlloc for proper memory handling.

modified on Friday, July 8, 2011 11:39 AM

GeneralRe: Something wrongmemberWes Jones8 Jul '11 - 18:10 
Thanks for all the great info!
GeneralRe: Something wrongmemberSamuelTee20 Jul '11 - 23:31 
Thanks too. To be honest: I haven't understood a bit. But this sounds somewhat important. So will there be any update on the above linked demo/src projects?
GeneralRe: Something wrongmemberWes Jones21 Jul '11 - 5:20 
In my spare time, I'm going to work on an update to the code that will support the 6000 files at once test. Btw, this code as-is has passed the 1000 files at once test.
The new version will be unit tested w/ CppUnit and in a dll rather than source files so I can break the code out and refactor a bit more cleanly. Of course, all that takes time, but it's someting I want to do.
GeneralRe: Something wrongmember_stefanu_7 Jul '11 - 1:13 
After some more tests, I am now pretty convinced that depending on how fast files are created and how long On_FileAdded takes, some events are missed. Still not sure about the "why", but I think this has something to do with how many events are returned in a call to RDCW, and how many other files are created before the next call, as you already pointed out.
 
Maybe a synchronous call would to the trick; still working on this.
GeneralMy vote of 4memberMember 78963284 May '11 - 5:26 
good
Generalwhat about 'copy' eventmemberbatsword3 May '11 - 19:04 
i can't get 'copy event '
ReadDirectoryChangesW can't do this ?
what about hook
GeneralMy vote of 5memberSk0rp12 Mar '11 - 2:03 
Very good. Thanks.
GeneralCompile in MS VS 2008memberSk0rp12 Mar '11 - 2:01 
For cimpile in VS 2008 need add:
friend class CDelayedDirectoryChangeHandler;
in
declaration of class CDirectoryChangeHandler
GeneralC++BuildermemberCmdrRicK4 Mar '11 - 1:36 
Any chance to compile this under C++Builder (BCB) ?
Compiler stops at the first CString.
CCriticalSection and other C... classes seem to be further problems.
GeneralRe: C++BuildermemberWes Jones4 Mar '11 - 7:00 
this uses the microsoft MFC library, which is where CString, CCriticalSection come from..... does borland support that?
GeneralRe: C++BuildermemberCmdrRicK4 Mar '11 - 20:40 
I know no way to use mfc in c++builder, seems I need to try on my own :(
GeneralMy vote of 5memberwebphoenix26 Dec '10 - 16:54 
It's so good and useful code,Thanks a lot!
GeneralI have a problem about the "ProcessChangeNotifications" function.memberyuanjuzeng19 Oct '10 - 4:14 
hi,wes jones.
If i modify the code like this, it also works.
void CDirectoryChangeWatcher::ProcessChangeNotifications()
{
   .....
   // if I modify here  
   // CDirectoryChangeHandler * pChangeHandler = pdi->GetChangeHandler();
   CDirectoryChangeHandler * pChangeHandler = pdi->GetChangeHandler()->GetRealChangeHandler();
   ....
}
Now,the Change Notification would be processed in the thread that monitors directory changes.
 
I want to know the reason that you use another worker thread to process the change notification.
I feel sorry for my poor english.I hope you can understand that and give me an answer.Thanks.
GeneralRe: I have a problem about the "ProcessChangeNotifications" function.memberWes Jones19 Oct '10 - 5:44 
Please don't change the internal code. It works & you only need to worry about the public interface.
 
It uses a background thread so that there is no window of time where a file notification can be missed. When ReadDirectoryChangesW returns, file notifications will be missed until ReadDirectoryChangesW is called again. It is important to call ReadDirectoryChangesW again as soon as possible. It is also important that the thread that processes notifications is not the same one that calls ReadDirectoryChangesW. All of this is so that file notifications are not missed in between calls to ReadDirectoryChangesW.
 
If you are using an app w/ a GUI, use true in the constructor. File notifications will occur on the same thread on which the instance of CDirectoryChangeWatcher was created.
Passing false will make them happen on a new background thread.
 
Good luck & thanks for using my code!
QuestionReadDirectoryChangesW does not provide specific modification informationmemberwinuser20026 Sep '10 - 20:49 
Hi,
 
I'm working on an application which monitors a directory using win api ReadDirectoryChangesW() . When monitoring the directory, this api sends FILE_ACTION_MODIFIED event in case a file/sub-directory is modified in the monitored directory. It doesn't tell what specifically has changed in the file or sub-directory since the time the file/ sub-directory entry was created in the monitored directory e.g size, timestamp or attributes or am I missing something. Is there any way to get what exactly has been modified viz size/timestamp/attirbutes since the time the file entry was created?
 
Thanks
Raj

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Permalink | Advertise | Privacy | Mobile
Web01 | 2.6.130523.1 | Last Updated 12 May 2002
Article Copyright 2001 by Wes Jones
Everything else Copyright © CodeProject, 1999-2013
Terms of Use
Layout: fixed | fluid