Click here to Skip to main content
Click here to Skip to main content

Directory Mirror using the FileSystemWatcher class

By , 31 Mar 2013
Rate this:
Please Sign up or sign in to vote.

About

The Directory Mirror is an experimental application that uses and extends the Microsoft .NET framework FileSystemWatcher class. It monitors the files and sub-folders of a specified source directory and maintains a copy of it in another directory. This application can help you learn things about the FileSytemWatcher class and IO file and directory operations.

Things to know about the FileSystemWatcher

The first thing to keep in mind when working with the FileSystemWatcher is that it's not a one for one relationship between the IO actions in the monitored directory and the events raised by the FileSystemWatcher. For an application like this one you will quickly notice that the created event is always followed by at least one changed event. When a file is copied into a monitored directory, it is first created (the header and info is first written) and then the actual data is written. Depending on the file size (and the time it takes to write it, as well as system performance and availability), many changed events may be raised because the file is constantly changing as it is being written (monitored by the LastWrite and/or Size NotifyFilters of the FileSystemWatcher) and this may vary greatly from one system/environment to another.

The internal buffer size is limited to a maximum value of 64KB but for an application like this one isn't not the bottleneck. A big batch operation can send a lot of information to the FileSytemWatcher so fast that it wont be able to keep track, long before the buffer is full; You will get the to many changes at once error. For instance if you select a few hundred files in a monitored directory and then hit delete, it will probably happen. The same thing can happen if you move a large number of files to the monitored directory and those files are located on the same partition as the monitored directory; Being on the same partition, no actual copying really happens, only the address of the files change and this happens very fast.

The solution

The solution contains two projects, one for the forms and the other for the Directory Mirror class and things realted to that class. I won't go through the forms here but just explain what's in the FSWDirectoyMirror project. First there's a folder containing some static IO and XML methods. The controller sits between the forms and the directory mirror objects. There's the DirectoryMirror of course. The DMHolder is used to hold and help keep track of the configuration information of DirectoryMirror objects. Enum contains a few enums of course. FSWabstract is an abstract class based on the FileSystemWatcher class and The DirectoyMirror class is based on this. I could have skipped this and base The DirectoryMirror class directly on the FileSystemWatcher but I often like to use an abstract as a starting point because it reminds me how I thought things at the beginning and helps me keep things organized. MirrorEventArgs is an entity used to carry information from the events raised in the mirror folder. Configurations of the DirectoryMirror objects are written to the mirrors.xml file (if the file is not found the application creates a new one). SourceEventArgs is an entity used to carry information from the events raised in the source folder. The following diagram shows how the application is structured; there is no direct communication between the DirectoryMirror objects and the forms, everything passes through the controller object.

The screens

Main screen

At the top of the form there are 3 buttons, the info button displays info about the application, the add button to create a new configuration and the save button that writes the list of configurations to an XML file. You then have 4 checkboxes for displaying information in the activity tab:

  • View source activity: Reports activity detected in the source directory.
  • View source errors: Reports errors detected in the source directory
  • View mirror activity: Reports activity detected in the mirror directory.
  • View mirror errors: Reports errors detected in the Mirror directory
  These checkbox options can be changed while instances of the DirectoryMirror are running.

Configurations tab 

The configurations tab first shows a list of DirectoryMirrors controls, one for each configuration. The name is displayed in the upper left corner followed by 3 buttons. The tool button opens the edit form to make changes. The recycle bin button deletes the configuration and the third one is the strart/stop button. Next are displayed the timer and buffer values. Under those values are displays the paths to the source directory and mirror directory.

The bottom part of the control offers various options:

  • InfoMode: No mirror directory is used and the application only reports about the activity detected in the source directory.
  • MirrorMode: In reaction to the changes detected in the source directory, the contents of the mirror directory are updated to match.
  • Monitor changed event: Enables or disables the tracking of the FileSystemWatcher's "changed" event.

Activity tab

The first grid shows the activity in the monitored directory and the second one is the mirror directory. Another way of saying it would be that the first grids shows the FileSystemWatcher's native events and the second grid pertains to the events we've added to the FileSystemWatcher to make it a DirectoryMirror.

Edit screen

The timer indicates how much time to wait before updating the mirror folder.

The buffer is the internal buffer size of the FileSystemWatcher object. It stores information about the detected events and the files and directories they pertain to. The buffer can overflow if it has more information that it can handle. The buffer comes from non-paged memory and can not be swapped out to disk.

The name textbox let's you set a friendly name to identify the configuration. The source and mirror textboxes and buttons are for selecting or typing these paths. For a remote computer you must use UNC paths (i.e.,: \\remoteComputer\targetFolder), mapped drives and removable USB storage will not work.

All fields are required and a validation is performed before you can confirm changes and return to the list of configurations. There is also a validation to ensure that the mirror directory isn't contained in the source directory and vice versa this would start a catastrophic reaction. However if you are running multiple DirectoryMirrors there is no validation of that sort between the running DirectoryMirrors; you could start an instance that has folder A as the source and folder B as the mirror and another one with folder B as the source and folder A as the mirror. You don't want to do that so use caution when running multiple instances.

Rebuild dialog

In order for this application to work, the contents of the source and mirror folders must be identical when the application starts. When starting an instance of the DirectoryMirror you have three options to rebuild the mirror:

  • Hard rebuild deletes the contents of the mirror and copies the contents of the source into it.
  • Comparative rebuild compares the contents of the two folders by file name and size and makes the appropriate changes ( a good option if you don't have a lot of files but some very big ones).
  • Delete all deletes the contents of both folders.

How it works 

Every time an events is fired in the source folder, the timer is reset and information about the event is sent to a queue. When the timer expires, the events from the queue are copied to a list, the contents of the queue are cleared and the process of updating the mirror folder begins, using the newly created list. This way new events can be sent to the queue while the mirror is being updated. The timer does not guaranty that there is no more activity in the source folder when it's time to update the mirror but it diminishes that possibility. If you can anticipate what kind of file activity you will get you can adjust the timer value accordingly.

You’ll notice there is a checkbox to let you monitor or not the changed event. Because it only occurs after a created event, it is not necessary to monitor it for this application. The application works pretty good but if you have some intense file activity (big batch operations involving hundreds of files and/or some very large files).  As mentioned earlier the firing of events by the FileSystemWatcher can differ depending on the system/environment. Try with and without the changed event and using a bigger timer value.

The code

The FSWabstract abstract class

using System;
using System.IO;

namespace DM
{
    // This class is based on the FileSystemWatcher class
    public abstract class FSWabstract : FileSystemWatcher
    {
        // Delegates and events to send messages about FileSystemWatcher events
        public delegate void SourceEventDelegate(SourceEventArgs fswEventArgs);
        public event SourceEventDelegate SourceEvent;
        public delegate void SourceErrorDelegate(SourceEventArgs fswEventArgs);
        public event SourceErrorDelegate SourceError;

        public string FriendlyName { get; set; }

        protected FSWabstract()
        {
            // Changed, Created and Deleted event handlers have common parameters "object"
            // and "FileSystemEventArgs" so we'll use the same delegate for those 3
            Changed += FSWcontract_IoActivity;
            Created += FSWcontract_IoActivity;
            Deleted += FSWcontract_IoActivity;
            Renamed += FSWcontract_Renamed;
            Error += FSWcontract_Error;
        }

        // Enable or disable tracking of "Changed" event.
        public void TrackChangedEvent(bool value)
        {
            if (value) Changed += FSWcontract_IoActivity;
            else Changed -= FSWcontract_IoActivity;
        }

        // Tracks FilsSystemWatcher Errors
        private void FSWcontract_Error(object sender, ErrorEventArgs e)
        {
            SourceEventArgs fswe = new SourceEventArgs();
            fswe.FSWname = this.FriendlyName;
            fswe.TimeStamp = DateTime.Now;
            fswe.Path = e.GetException().Message;
            fswe.EventType = "Error";
            SourceError(fswe);
        }

        // Tracks FilsSystemWatcher "Renamed" events
        private void FSWcontract_Renamed(object sender, RenamedEventArgs e)
        {
            SourceEventArgs fswe = new SourceEventArgs();
            fswe.FSWname = this.FriendlyName;
            fswe.TimeStamp = DateTime.Now;
            fswe.OldPath = e.OldFullPath;
            fswe.Path = e.FullPath;
            fswe.EventType = e.ChangeType.ToString();
            SourceEvent(fswe);
        }

        // Tracks FilsSystemWatcher "Changed", Created" and "Deleted" events
        private void FSWcontract_IoActivity(object sender, FileSystemEventArgs e)
        {
            SourceEventArgs fswe = new SourceEventArgs();
            fswe.FSWname = this.FriendlyName;
            fswe.TimeStamp = DateTime.Now;
            fswe.Path = e.FullPath;
            fswe.EventType = e.ChangeType.ToString();
            SourceEvent(fswe);
        }
    }
}

The DirectoryMirror class

using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Timers;
using DM.StaticMethods;

namespace DM
{

    // This class is based on the FSWabstract abstract class
    public class DirectoryMirror : FSWabstract
    {
        string _mirDir = ""; // Path of the mirror directory
        List<sourceeventargs> _IOlist; // List to keep track of chages in the source diretory
        double _time;
        Timer _timer;
        bool _mirrorMode;

        public string SourceDirectory
        {
            get { return this.Path; }
            set { this.Path = value; }
        }

        public string MirrorDirectory
        {
            get { return _mirDir; }
            set { _mirDir = value; }
        }

        public double Milliseconds
        {
            get { return _time; }
            set { _time = value; }
        }
        public bool MirrorMode
        {
            get { return _mirrorMode; }
            set { _mirrorMode = value; }
        }        

        //Delegates and events to send messages about Exceptions
        public delegate void MirrorActionDelegate(MirrorEventArgs info);
        public event MirrorActionDelegate DMinfoEvent;
        public delegate void MirrorErrorDelegate(MirrorEventArgs info);
        public event MirrorErrorDelegate DMerrorEvent;

        public DirectoryMirror()
        {
        }

        public DirectoryMirror(string name, string srcDir, string mirDir)
        {
            this.FriendlyName = name;
            _IOlist = new List<sourceeventargs>();
            // Set up the path to the source directory
            SourceDirectory = srcDir;
            // Set up the path to the mirror directory
            MirrorDirectory = mirDir;
            //Start();
        }

        public void Start()
        {
            if (!string.IsNullOrEmpty(SourceDirectory) &&
                !string.IsNullOrEmpty(MirrorDirectory))
            {
                // Set up the different properties
                // Monitor all files and directories
                Filter = "";
                // Listen for changes in the name of files and directories
                // and changes in size
                NotifyFilter = (NotifyFilters.FileName |
                    NotifyFilters.DirectoryName | NotifyFilters.LastWrite);
                IncludeSubdirectories = true;
                EnableRaisingEvents = true;


                if (_mirrorMode)
                {
                    SourceEvent += DirectoryMirror_FSWevent;
                    SourceError += DirectoryMirror_FSWerror;
                    _timer = new Timer();
                    _timer.Elapsed += timer_Elapsed;
                    _timer.Interval = _time;
                    _timer.AutoReset = false;
                    _IOlist = new List<sourceeventargs>();
                }
            }
        }

        void DirectoryMirror_FSWerror(SourceEventArgs fswEventArgs) 
        {
            _timer.Start();
        }

        void DirectoryMirror_FSWevent(SourceEventArgs fswEventArgs) 
        {          
            _timer.Start();
            _IOlist.Add(fswEventArgs);
        }

        private void timer_Elapsed(object sender, ElapsedEventArgs e)
        {
            MirrorEventArgs args = new MirrorEventArgs();
            args.FSWname = this.FriendlyName;
            args.TimeStamp = DateTime.Now;
            args.Info = "Timer elapsed";
            DMinfoEvent(args);
            if (_IOlist.Count > 0)
            {
                List<sourceeventargs> list = new List<sourceeventargs>();
                lock (_IOlist)
                {
                    list.AddRange(_IOlist);
                    _IOlist.Clear();
                }

                updateMirror(list);
            }
        }

        private void updateMirror(List<sourceeventargs> list)
        {
            MirrorEventArgs args = new MirrorEventArgs();
            args.FSWname = this.FriendlyName;
            args.TimeStamp = DateTime.Now;
            args.Action = "Begin Mirror Update";
            args.Info = list.Count.ToString() + " event(s) to process";
            DMinfoEvent(args);

            string destination;
            List<sourceeventargs> listF = new List<sourceeventargs>();

            try
            {
                //--- Created ---//
                listF = list.Where(x => x.EventType == "Created").ToList();
                for (int i = 0; i < listF.Count; i++)
                {
                    destination = listF[i].Path.Replace(SourceDirectory, MirrorDirectory);                    
                    if (Directory.Exists(listF[i].Path))
                    {                       
                        Directory.CreateDirectory(destination);
                        IOmethods.CopyDirectoryRecursively(listF[i].Path, destination);
                        list.Remove(listF[i]);
                        // Remove all "Created" and "Changed"
                        // events of child folders and files from the master list
                        list = list.Where(s => (s.EventType == "Created" || 
                          s.EventType == "Changed") && s.Path.Contains(listF[i].Path) == false).ToList();
                        // Update the sublist of "Created events"
                        listF = list.Where(x => x.EventType == "Created").ToList();
                        i -= 1;
                        args = new MirrorEventArgs();
                        args.FSWname = this.FriendlyName;
                        args.Action = "Create";
                        args.TimeStamp = DateTime.Now;
                        args.Info = destination;
                        DMinfoEvent(args);

                    }
                    else
                    {
                        File.Copy(listF[i].Path, destination, true);
                        // Remove all "Changed" events for this file from the master list
                        list = list.Where(s => (s.EventType != "Changed" && s.Path != listF[i].Path)).ToList();
                        // Update the sublist of "Created events"
                        listF = list.Where(x => x.EventType == "Created").ToList();
                        i -= 1;
                        args = new MirrorEventArgs();
                        args.FSWname = this.FriendlyName;
                        args.Action = "Create";
                        args.TimeStamp = DateTime.Now;
                        args.Info = destination;
                        DMinfoEvent(args);
                    }
                }
            }
            catch (Exception x)
            {
                args = new MirrorEventArgs();
                args.FSWname = this.FriendlyName;
                args.TimeStamp = DateTime.Now;
                args.Action = "Error";
                args.Info = x.Message;
                DMerrorEvent(args);
            }
            try
            {
                //--- Changed ---//
                listF = list.Where(z => z.EventType == "Changed").ToList();
                listF = RemoveDuplicates(listF);
                foreach (SourceEventArgs f in listF)
                {
                    if (File.Exists(f.Path))
                    {
                        destination = f.Path.Replace(SourceDirectory, MirrorDirectory);
                        File.Copy(f.Path, destination, true);
                        args = new MirrorEventArgs();
                        args.FSWname = this.FriendlyName;
                        args.Action = "Copy";
                        args.TimeStamp = DateTime.Now;
                        args.Info = destination;
                        DMinfoEvent(args);
                    }

                }
            }
            catch (Exception x)
            {
                args = new MirrorEventArgs();
                args.FSWname = this.FriendlyName;
                args.TimeStamp = DateTime.Now;
                args.Action = "Error";
                args.Info = x.Message;
                DMerrorEvent(args);
            }
            try
            {
                //--- Renamed ---//
                listF = list.Where(x => x.EventType == "Renamed").ToList();
                foreach (SourceEventArgs f in listF)
                {
                    destination = f.Path.Replace(SourceDirectory, MirrorDirectory);
                    if (System.IO.Directory.Exists(f.OldPath.Replace(SourceDirectory, MirrorDirectory)))
                    {
                        string oldFPath = f.OldPath.Replace(SourceDirectory, MirrorDirectory);
                        string newFPath = f.Path.Replace(SourceDirectory, MirrorDirectory);
                        System.IO.Directory.Move(oldFPath, newFPath);
                    }
                    else
                    {
                        string oldFPath = f.OldPath.Replace(SourceDirectory, MirrorDirectory);
                        string newFPath = f.Path.Replace(SourceDirectory, MirrorDirectory);
                        System.IO.File.Move(oldFPath, newFPath);
                    }
                    args = new MirrorEventArgs();
                    args.FSWname = this.FriendlyName;
                    args.Action = "Rename";
                    args.TimeStamp = DateTime.Now;
                    args.Info = destination;
                    DMinfoEvent(args);
                }
            }
            catch (Exception x)
            {
                args = new MirrorEventArgs();
                args.FSWname = this.FriendlyName;
                args.TimeStamp = DateTime.Now;
                args.Action = "Error";
                args.Info = x.Message;
                DMerrorEvent(args);
            }
            try
            {
                //--- Deleted ---//
                listF = list.Where(x => x.EventType == "Deleted").ToList();
                foreach (SourceEventArgs f in listF)
                {
                    destination = f.Path.Replace(SourceDirectory, MirrorDirectory);
                    if (Directory.Exists(destination))
                    {
                        Directory.Delete(destination, true);
                    }
                    else
                    {
                        File.Delete(destination);
                    }
                    args = new MirrorEventArgs();
                    args.FSWname = this.FriendlyName;
                    args.Action = "Delete";
                    args.TimeStamp = DateTime.Now;
                    args.Info = destination;
                    DMinfoEvent(args);
                }
            }
            catch (Exception x)
            {
                args = new MirrorEventArgs();
                args.FSWname = this.FriendlyName;
                args.TimeStamp = DateTime.Now;
                args.Action = "Error";
                args.Info = x.Message;
                DMerrorEvent(args);
            }
            args = new MirrorEventArgs();
            args.FSWname = this.FriendlyName;
            args.TimeStamp = DateTime.Now;
            args.Action = "End Mirror update";
            DMinfoEvent(args);
        }

        private List<sourceeventargs> RemoveDuplicates(List<sourceeventargs> list)
        {
            list = list.OrderBy(z => z.FSWname).OrderBy(z => z.Path).ToList();
            List<sourceeventargs> newList = new List<sourceeventargs>();
            SourceEventArgs last1 = new SourceEventArgs();
            last1.Path = "+";
            last1.FSWname = "";
            foreach (SourceEventArgs fswa in list)
            {
                if (fswa.FSWname != last1.FSWname || fswa.Path != last1.Path)
                {
                    newList.Add(fswa);
                    last1 = fswa;
                }
            }
            return newList;
        }
    }
}

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Luc Archambault
Web Developer
Canada Canada
I have been working with different web technologies (ASP, ASP.NET, ColdFusion, Perl/CGI) for the past 15 years. I live in Montreal, where I work for a little consultant group specializing in Microsoft technologies.

Comments and Discussions

 
QuestionDirectory Mirror Pinmemberdoctorcpu29-Apr-13 6:15 
AnswerRe: Directory Mirror PinmemberLuc Archambault1-May-13 8:49 
QuestionNice work, and two questions, and one suggestion PinmemberBillWoodruff29-Mar-13 3:33 
AnswerRe: Nice work, and two questions, and one suggestion [modified] PinmemberLuc Archambault31-Mar-13 14:31 
GeneralMy vote of 5 PinmemberWhiteOsoBDN4-Mar-13 5:20 
GeneralRe: My vote of 5 PinmemberLuc Archambault5-Mar-13 16:39 
QuestionVery nice work. PinmemberJipin9-Apr-12 20:23 
AnswerRe: Very nice work. PinmemberLuc Archambault10-Apr-12 10:31 
GeneralRe: Very nice work. PinmemberJipin11-Apr-12 7:15 
GeneralRe: Very nice work. PinmemberLuc Archambault20-Apr-12 12:38 
Questioncan u provide the document for this project PinmemberMember 823648420-Sep-11 0:51 
AnswerRe: can u provide the document for this project PinmemberLuc Archambault20-Sep-11 8:11 
QuestionHow to get closing event of other application in my running application PinmemberPatel Pranav23-Apr-10 20:32 
AnswerRe: How to get closing event of other application in my running application PinmemberLuc Archambault27-Apr-10 12:40 
GeneralThank you Pinmembernirvansk8157-Apr-08 15:28 
GeneralWorks great, but we need a few tweaks. Pinmemberlmeyers15-Feb-08 17:04 
GeneralEvents firing twice Pinmemberp10006-Mar-07 3:45 
Generalnice work ~!great Pinmembernicesnow5-Feb-07 15:57 
Questionhow bout web application? Pinmember!ndera12-Dec-06 16:51 
AnswerRe: how bout web application? PinmemberLuc Archambault13-Dec-06 7:51 
GeneralMonitor a list of path Pinmemberpasdan22-May-06 22:41 
QuestionHow can we solve the IO exception? Pinmemberchallengerking19-Apr-06 17:06 
AnswerRe: How can we solve the IO exception? PinmemberLuc Archambault19-Apr-06 18:45 
GeneralRe: How can we solve the IO exception? Pinmemberchallengerking19-Apr-06 19:32 
GeneralRe: How can we solve the IO exception? Pinmemberchallengerking20-Apr-06 16:11 
AnswerRe: How can we solve the IO exception? PinmemberFayyaz Lodhi3-Jan-07 19:21 
AnswerRe: How can we solve the IO exception? Pinmembersides_dale14-Apr-07 20:23 
GeneralRe: How can we solve the IO exception? PinmemberFayyaz Lodhi15-Apr-07 20:28 
GeneralRe: How can we solve the IO exception? Pinmembersides_dale22-Apr-07 14:28 
GeneralRe: How can we solve the IO exception? PinmemberEpaxx15-Aug-07 11:28 
GeneralNice work Luc Pinmemberwconnors14-Nov-05 4:52 
GeneralDoesn't work for VS 2005 Beta 2 anymore PinmemberMike Wild14-Jun-05 3:45 
GeneralRe: Doesn't work for VS 2005 Beta 2 anymore Pinmemberflatsheep200518-May-06 11:32 
GeneralThank you Luc! PinsussAnonymous24-Feb-05 20:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140415.2 | Last Updated 31 Mar 2013
Article Copyright 2005 by Luc Archambault
Everything else Copyright © CodeProject, 1999-2014
Terms of Use
Layout: fixed | fluid