Click here to Skip to main content
Click here to Skip to main content
Go to top

Eyes on NTFS

, 9 Sep 2005
Rate this:
Please Sign up or sign in to vote.
An introduction to NTFS's Change Journal and a simple tool to operate
Sample Image

Introduction

The Change Journal is a database that contains a list of every change made to the files or directories on an NTFS 5.0 volume, that is, the Change Journal is a log file.

The structure of Change Journal's record includes USN, Reason, Filename, ParentFileReferenceNumber and TimeStamp, etc.
Let me explain them to you:

  • USN is ID
  • Reason is why the record is written, say, 'create'
  • Filename is merely name without path
  • ParentFileReferenceNumber is a foreign key to file's path
  • TimeStamp is the time when event happens

With this information, we can know what happened to a file, the only problem is where to find the path of the file. There is no simple/easy way to get there as far as I know. Let's face it, NTFS is not a RDBMS, we can't use one 'select' sentence from relevant tables to get all we need. Although, in concept, ParentFileReferenceNumber is a foreign key. Well, what Window2000 provides us to get a FileReferenceNumber is using an opened handle to the file, the API is GetFileInformationByHandle. We have to search the whole directory tree to find what we want! -- That's why I feel tired for hard drive every time I push the 'Path detail' button.

The other thing we should know is that the Change Journal is not available automatically. Somebody should activate/create it before using it.

To archive functions on Change Journal, the key API is DeviceIoControl. To fulfill a retrieval of the Change Journal records, the basic steps are: 

  1. Open the Change Journal file handle using API CreateFile 
  2. Call DeviceIoControl using FSCTL_QUERY_USN_JOURNAL as the second parameter 
  3. Call DeviceIoControl using FSCTL_READ_USN_JOURNAL as the second parameter

Step (1) opens the handle and the handle is also a parameter needed by step (2); Step (2) gets the general information about the Change Journal, the key is UsnJournalID and FirstUsn. UsnJournalID is Session ID, which is needed by every call of Step (3). FirstUsn is needed by first time call of Step (3). Step (3) is called repetitively until we get all the records. Every call of Step (3) requires a StartUsn which indicates from which record we are searching and it could be fetched from the previous call, is it simple for your loop?

The demo code includes a class CChangeJournal and its consumer which is a dialog. CChangeJournal is used to operate the Change Journal, including creating and disabling(deleting) it.

Note: I learned all this from MSDN, especially from the article: 'Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained ' by Jeffrey Cooperstein and Jeffrey Richter. In addition, a lot of the code in this demo is from their demo. I simplified the code by ignoring the OVERLAPPED I/O.

Background

Sometimes, you may have done a series of operations on files and probably one of them is wrong, but after a short time, you couldn't remember where the file is and you don't want to roll back all the operations by using 'Undo' provided by the Explorer, because most operations are what you want. At such times, this tool might help.
Anyway, some real time anti-virus software may use this technique.

Using the Code

CChangeJournal provides some functions according to the steps I mentioned above. Let me explain with the code below:

CChangeJournal cj('c');              	// 'c' represents "C:" ;
cj.open();                              	// open file handle    [step 1]

cj.query(...);                           	// get general information, 
				    	// key is UsnJournalID [step 2]

CJRecords recs;                        	// CJRecords is a vector of part 
				    	// structure of Change Journal

cj.getRecords(startUsn,..,recs,...);     	// get records we need [step 3]

// the other things
cj.create();                             	// create Change Journal

cj.disable(UsnJournalID);                	// delete Change Journal need a parameter 
				    	// from (step 2: query)

cj.setDriver('D');                       	// change Drive

cj.setMemoryBlock(bytes);                	// how large you want to allocate for 
				    	// reading records. it's called
                                         	// with default 1M size, you can change 
				    	// if you like. 

Points of Interest

There are several things I want to mention:

  1. There is another way to fetch records by using FSCTL_ ENUM_USN_DATA as the parameter of DeviceIoControl, find out from MSDN.
  2. In my demo, within the dialog(consumer), I set 'nextUsn' not the real next Usn, while the current last Usn, so I get one duplicate record every time (except page 1), but it prevents calling being blocked. This is because the query will be blocked if the Next usn hasn't been created. Unfortunately, I couldn't stop it being blocked when there is nothing in the database when the Change Journal is just being created!
  3. Firstly when I tested my demo, I could find all the paths associated with the ParentFileReferenceNumber except the root directory, in my case, say "C:". Then I tried "C:\", o.k., it started working. It bothers me so far. Could you tell me why when you find out, thanks a lot.
  4. What if a directory is renamed? It doesn't change its FileReferenceNumber; If we delete a directory, of course we can't find it any more, because we have no way to get its opened handle. Then what if we recreate a directory with the same name? The FileReferenceNumber changes! So, if we really want to keep an eye on NTFS, we should maintain a database of our own to trace every change we make -- just as the two Jeffreys do in their demo.
  5. What's the rule of a new FileReferenceNumber? I couldn't find it.

History

  • Article created on 9th September 2005

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Jason Q Zhang
Web Developer
Australia Australia
6 years developing on Windows, C++, RDBMS, Win32. STL is my favorite. Like really cool techniques

Comments and Discussions

 
QuestionHow to find number of files and folders in any specified directory using USN Journal [modified] Pinmemberajaytvish25-Sep-13 2:02 
GeneralMy vote of 2 Pinmembercocowalla11-May-11 3:06 
GeneralNever see USN_REASON_FILE_DELETE Pinmemberteubies7-Apr-11 2:43 
GeneralMessage Removed Pinmemberegl10441-May-11 3:52 
GeneralRe: Never see USN_REASON_FILE_DELETE Pinmemberteubies1-May-11 21:28 
Generalcomplete rollback Pinmembervvurade26-Feb-11 19:28 
Generalsome wrong on 2t disk simuator Pinmembernhchmg15-Dec-10 22:56 
QuestionHow to Read File Path and Last USN Number PinmemberGul Zakhmi29-Nov-07 18:04 
GeneralErrors while Compiling\linking PinmemberMIAN KAMRAN10-Sep-06 23:20 
GeneralRe: Errors while Compiling\linking PinmemberMIAN KAMRAN10-Sep-06 23:25 
GeneralRe: Errors while Compiling\linking PinmemberJason Q Zhang11-Sep-06 20:46 
GeneralReading a remote File change journal Pinmembervikijain8-May-06 12:28 
GeneralReadDirectoryChangesW Pinmemberumeca7413-Sep-05 1:28 
GeneralRe: ReadDirectoryChangesW PinmemberJason Q Zhang13-Sep-05 18:07 
GeneralRe: ReadDirectoryChangesW [modified] Pinmembermarc ochsenmeier27-Jun-10 22:21 
GeneralNot active PinmemberSpole11-Sep-05 6:22 
GeneralRe: Not active PinmemberJason Q Zhang11-Sep-05 17:59 
GeneralFiles missing PinmemberK-ballo9-Sep-05 22:36 
GeneralTemporary links PinmemberRavi Bhavnani10-Sep-05 3:03 
GeneralRe: Temporary links PinmemberJason Q Zhang11-Sep-05 2:55 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140926.1 | Last Updated 10 Sep 2005
Article Copyright 2005 by Jason Q Zhang
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid