File System Notification Kit (FSNK) Introduction

Introduction

File System Notification Kit (FSNK) is a developers product for monitoring file system activity in real-time.

If you want to know what’s happening in the file system, you don’t need to spend a mammoth amount of time writing a driver. Just download and try the FSNK.

FSNK uses a kernel-mode driver, which is a file system filter. When the filter receives a file system request notification, it sends the information about it to the application via the callback function specified by the application when connecting to the FSNK engine.

The engine is able to view all the basic file operations, such as writing and reading data in/from a file, modifying file or folder attributes, creating files or folders, deleting or moving files, and so on. When you receive a notification, you can get more detailed information about the event.

In addition to the standard file operations, the kit can track volume mount and unmount operations, security descriptor modifications, object ID changes, cache flush operations, creation of hard and symbolic links, and other file system events.

FSNK Contents

Description of files in the FSNK installation package:

• Fsnk.sys is a kernel-mode driver that intercepts file system operations. It can be loaded and unloaded dynamically without system rebooting.
• Fsnklib.dll is a dynamic link library that allows applications to interact with the driver and manage its state. It provides the file system activity information for your application.
• Fsnkd.exe is a console utility that demonstrates the product features.
• Install.exe is a simple application for installing and uninstalling the product.
• Fsnklib.lib is a static library for linking to the dynamic library at the compile time.
• *.h-files are header files containing definition of API functions and data types intended for C/C++ projects.

Requirements

Currently, the product supports all versions of Windows from XP SP2 to Server 2012 R2, including 64-bit systems for x86-64 architecture (a build for Itanium-based systems is also available upon request). The SP2 update is required due to the Filter Manager component, which is necessary for fsnk.sys and cannot be installed separately. Server systems require Windows Server 2003 with SP1 or a later version.

Features

File System Notification Kit gives you a great opportunity to understand what is going on in the file system at a given time. It provides notifications on all file operations: creating files, deleting, reading, writing, reading and modifying file metadata, security descriptors, attributes, etc.

FSNK supports both familiar DOS-style paths C:\Windows, and NT-style paths \Device\HarddiskVolume1\Windows, which are native to the operating system.

Using FSNK, you will receive detailed information on each file operation. This information includes the operation code, operation status, unique thread, process and session IDs, logon session ID, security identifier (SID) of the user, file object volume ID, full normalised NT-style name of the file system object and offset to the relative file path in characters.

FSNK collects statistics and information about the file system performance, including the number of bytes read and written, number of read and write operations, etc. It has a built-in notification filter for receiving notifications about the events of your interest only.

For each file system event, you can call the FsnkGetName (and FsnkGetTargetName if applicable) function to obtain the DOS-style name of the file system object associated with the operation. You can also call the FsnkGetAccountName routine to get the name of the user who has requested the operation. And you can call the FsnkGetProcessImagePath routine as well to get the path to the executable file of the process initiating the operation.

It provides a set of API functions for file I/O at a level below Win32 subsystem and Native layer. This allows you to bypass potential interceptors at these levels. The FsnkIoCreate function can open and create file objects with the use of NT-style file name. The other functions in the group enable you to work with files using the received handle.

FSNK works equally well with any file system that is supported by the operating system, including disk file systems, CDs and even network file systems.

Please note that the current release of FSNK cannot block operations or change their parameters. At this point, it can only provide the notifications about the events already occurred.

API

This section further describes the API functions of the basic functionality of the product.

First, you need to initialise the engine via the FsnkInitialize API function; you cannot call other API functions of FSNK before this is done. You should pay attention that only one process at a time can be connected to the engine. It means if one of your applications connected to the engine to get a notification, another application will not be able to do this. To de-initialise the engine and release all the resources used by the library, you should call the FsnkUninitialize function.

Installation

The FsnkInstall function installs a kernel-mode filter driver (fsnk.sys). The input parameters include the service name that will be displayed in the system services list, the full path to the driver file, the flag responsible for automatic startup of the driver during the system boot, and Altitude ID. This API function returns ‘true’ if the engine installation was successful, or ‘false’ if not.

// Installing the engine.
if (! FsnkInstall (
    L"My FSNK engine",
    L"C:\\Program Files\\My product\\fsnk.sys",
    TRUE,        // For automatic startup.
    L"265000",    // Specify your own altitude here.
    0))        // No flags defined at this moment.
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

The thread that is calling the function must have administrative rights for successful engine installation. Do not forget to obtain the Altitude ID from Microsoft if you want to use FSNK in your product for mass distribution. The Altitude ID should be requested from the FSFilter Activity Monitor group, because the monitoring filters should work at this level.

The FsnkUninstall API function uninstalls the engine but does not unload it from memory. You should either previously stop the engine via the FsnkUnload routine, or restart the system to remove the engine completely. The engine can be started and stopped as many times as you wish without system reboots. The function returns ‘true’ if the operation was successful, or ‘false’ if not.

// Remove the engine from the system.
if (! FsnkUninstall ())
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

Similarly to the installation, this function must be called from the thread running in the context of an administrator user.

Managing filter state

The FsnkLoad API function starts the engine. Startup is done without any parameters. The function returns ‘true’ if the startup was successful, or ‘false’ if not. After this, the application can connect to the engine via FsnkInitialize. To unload the engine from memory, call the FsnkUnload function.

// Load the engine, this will actually load the FSNK filter driver.
if (! FsnkLoad ())
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

// Activate filter to start receiving notifications about file system events.
if (! FsnkActivate ())
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
    // Perform any cleanup needed.
    FsnkUnload ();
    ...
}

The FsnkActivate API function turns on the sending of notifications about file system events from the filter to the application through a callback routine whose address was specified in call to FsnkInitialize. After this, it becomes possible to receive events from the filter. Note that your callback function should be ready to receive notifications before the filter is activated. To temporary disable the filter, call the FsnkDeactivate function.

// Stop receiving notifications.
if (! FsnkDeactivate ())
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

// Unload FSNK filter from memory.
if (! FsnkUnload ())
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

Managing notification filter rules

The filters of the events you want to receive notifications about can be set and removed via FsnkAddRule and FsnkDeleteRule API functions. These functions insert and remove the rules from the rule list that is used for filtering file system events. The filters are set with various parameters described in the function input parameters. Initially, no filters are set and the application will receive the notifications about all supported file operations.

Each rule must contain at least a non-zero operation bitmask, while the name pattern of the file system object and process ID are optional (in order to specify ‘any’ process ID, specify the FsnkPidInvalid value in the corresponding parameter). If the specified object name pattern is already in the rule list, the specified bitmask and process ID will replace this rule and no new rules will be created.

The uFlags parameter sets various flags to control function behaviour. For example, you can specify the FsnkRuleFlagSave flag if you want the rule to be stored in the rules database located in the registry; rules in this database are loaded automatically upon engine startup. By default, rules are stored only in memory and are not saved after the system or engine restart.

// Include all executable files.
if (! FsnkAddRule (
    L"*.exe",
    0,            // 0 = all operations.
    FsnkPidInvalid,        // Means all processes.
    FsnkRuleFlagSave))    // Store rule permanently.
{ 
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}
...

// Exclude temporary files.
if (! FsnkAddRule (
    L"*.tmp",
    FsnkOpRead | FsnkOpWrite,
    FsnkPidInvalid,
    FsnkRuleFlagExclude))    // Should not match to include.
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

The FsnkDeleteRule function removes a rule from the database. On input, the function takes the file object name pattern and parameter for the rule removal from the persistent database. The FsnkClearRules function removes all existing rules from the rule list or from persistent database.

// Delete rule for temporary files.
if (! FsnkDeleteRule (
    L"*.tmp",
    FsnkRuleDeleteActive))
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

// Delete all permanent rules.
if (! FsnkClearRules (
    FsnkRuleClearSaved))
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

File object names

The FsnkGetName function returns the full DOS-style name of the file system object used for the execution of the operation, e.g. the name of the file or folder created, or the folder removed. The input parameters include the address of the structure that stores the operation data, or the flags that provide a network path with a name if the object is located on a network file system. At output, if the operation was successful, the function fills the FSNK_NAME_INFORMATION with the object name information. The function returns a non-zero value if successful, and zero otherwise.

The FsnkGetTargetName function returns the full name of the object, but unlike FsnkGetName, it returns the name of the target object. For example, the destination file name for move operation, or the name of a hard or symbolic link that is being created.

void
__stdcall
MyFsCallbackRoutine (
    IN PVOID pContext,        // Application-defined value.
    IN PFSNK_OP_DATA pData,        // General information of the request.
    IN PFSNK_OP_PARAMS pParams)    // Operation-specific information.
{
    ULONG uError = 0;
    FSNK_NAME_INFORMATION NameInfo = {0};
    FSNK_NAME_INFORMATION TargetNameInfo = {0};
    ...
    
    // Preallocate string buffers. You should do that before you call FsnkInitialize() routine so these strings can be passed here as context for performance purposes.
    FsnkCreateString (
        &NameInfo.Name,        // String structure to initialise.
        65554,            // Maximum possible size of a string.
        NULL, 0);        // Let the library to allocate new buffers.
    ...
    
    // Query DOS-style name for the indicated file system object.
    if (! FsnkGetName (
        pData,
        FsnkGetNameNetworkInfo, // Ask for the network path as well.
        &NameInfo))
    {
        // Error handling goes here.
        uError = FsnkGetErrorCode ();
        ...
    }
    
    // Check if a name string is present in the returned name information.
    if (NameInformation.IsName)
    {
        // Deal with name as required.
        MyLogWriteFileName (&NameInfo.Name);
        ...
    }
    
    // Check if a share string is present in the returned name information.
    if (NameInformation.IsShare)
    {
        // The specified object is in the network file system and we have the name of the share.
        MyLogWriteFileName (&NameInfo.Share);
        ...
    }
    
    // Query DOS-style name for the target object.
    // Only applicable to some file operations.
    if (! FsnkGetTargetName (
        pData,
        pParams,        // Required as source.
        FsnkGetNameNetworkInfo, // Ask for network path as well.
        &TargetNameInfo))
    {
        // Error handling goes here.
        uError = FsnkGetErrorCode ();
        ...
    }
    ...
    
    // Perform cleanup tasks.
    FsnkFreeName (&NameInfo);
    FsnkFreeName (&TargetNameInfo);
    ...
}

Managing strings

String management in the library is done via the FSNK_STRING structures. The FsnkCreateString initialises string structure, and FsnkDeleteString function deletes string structure where deletion means cleaning up the used resources if necessary. The FsnkCreateString function can both initialise the structure with a pre-allocated buffer and create a new buffer by itself.

The size of the string buffer in the FsnkCreateString function is specified in the uMaximumSize input parameter, in bytes. This parameter is required and cannot be NULL. The first parameter indicates the string structure that is being initialised. The pwInitialData parameter is a previously prepared buffer; if it is not NULL, the string is initialised to use this buffer, otherwise the function allocates a new buffer. The last parameter specifies the size of the buffer in bytes.

FSNK_STRING FileName = {0};
PWSTR pwFileName = L"C:\\Windows\\notepad.exe";
USHORT uBufferSize = (lstrlen (pwFileName) + 1) * sizeof (WCHAR);

// Initialise for the existing string buffer.
if (! FsnkCreateString (
    &FileName,
    uBufferSize,    // Total size of the buffer.
    pwFileName,    // No buffers will be allocated.
    0,        // Actual size of the string, to be calculated.
{
    // Error handling goes here. You can't call FsnkGetErrorCode because string routines don't use error codes (for performance).
    ...
}

The FsnkDeleteString function for deleting strings takes the address of the structure that needs to be cleaned up; whereas the string buffer will be released only if it was allocated by the library. For example, a buffer in a string that was initialised via FsnkInitString will not be freed, the application is responsible for that.

FSNK_STRING String = {0};
...

// Free resources used for the string.
FsnkDeleteString (
    &String);

User account names

The FsnkGetAccountName API function allows you to use a security identifier for obtaining the name of the associated user account. This may be a SID from the FSNK_OP_DATA structure or any other valid SID. At output, you will get the address of the Unicode string containing the current name for the account. This string should be deleted via FsnkFreeString after the work with it is completed.

Since the user account name can be changed at any time, you should not rely on it to identify the account. Use SID for this, and request the account name only when necessary. For example, if you need to show it to a user or write into the event log.

void
__stdcall
MyFsCallbackRoutine (
    IN PVOID pContext,        // Application-defined value.
    IN PFSNK_OP_DATA pData,        // General information of the request.
    IN PFSNK_OP_PARAMS pParams)    // Operation-specific information.
{
    ULONG uError = 0;
    SID_NAME_USE uNameUse = 0;
    PWSTR pwAccountName = NULL;
    ...
    
    // Query user account name.
    if (! FsnkGetAccountName (
        pData -> Sid,        // User account SID.
        &pwAccountName,        // Name string will be returned here.
        &uNameUse))        // Account name usage type.
    {
        // Error handling goes here.
        uError = FsnkGetErrorCode ();
        ...
    }
    
    // Use account name string.
    MyLogWriteString (pwAccountName);
    ...
    
    // Perform cleanup tasks.
    FsnkFreeString (&pwAccountName);
}

Obtaining the process image file name

The FsnkGetProcessImagePath function returns a full DOS-style path to the executable image file of the process. At input, you should specify the process ID that can be obtained from the ProcessId field of the FSNK_OP_DATA structure. At output, you will get the address of the Unicode string that contains the path to the process image file. The string returned must eventually be released via FsnkFreeString. See the product documentation for more information.

void
__stdcall
MyFsCallbackRoutine (
    IN PVOID pContext,        // Application-defined value.
    IN PFSNK_OP_DATA pData,        // General information of the request.
    IN PFSNK_OP_PARAMS pParams)    // Operation-specific information.
{
    ULONG uError = 0;
    PWSTR pwImageFilePath = NULL;
    ...
    
    // Query image file path for the requestor process.
    if (! FsnkGetProcessImagePath (
        pData -> ProcessId    // Unique requestor ID.
        &pwImageFilePath))    // Image path string will be returned here.
    {
        // Error handling goes here.
        uError = FsnkGetErrorCode ();
        ...
    }
    
    // Use process image path string.
    MyLogWriteString (pwImageFilePath);
    ...
    
    // Perform cleanup tasks.
    FsnkFreeString (&pwImageFilePath);
}

File system performance statistics

The FsnkGetStatistics function provides statistical data on the file system performance, such as the total number of reading, writing and opening/creating file operations, including both successful and failed, the total number of bytes successfully read and written, the start time of statistics collection, the time of the last reading and writing operations in the file system. To see the demonstration of this feature, use the fsnkd stat command in the console demo.

ULONG uError = 0;
FSNK_STATISTICS_DATA FsStatistics = {0};

// Query current statistics.
if (! FsnkGetStatistics (
    &FsStatistics))
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

// Write statistics to the log.
MyLogPrintString ("Bytes read: %I64u", FsStatistics.BytesRead);
MyLogPrintString ("Bytes written: %I64u", FsStatistics.BytesWritten);
...

Note that if during the library initialisation the flag is set to FsnkInitFullStats, the statistics will include all supported I/O operations. Otherwise, the statistics will be collected according to the current filter rules, if any were added to the rule list.

Low-level file input/output

The FsnkIoCreate API function creates or opens a file system object with a specified name. This object can be a file, folder or logical disk (volume). If the FsnkIoBypassFilters flag is used, no notifications will be generated in the library for open or create file operations executed via this function. In addition, this flag allows you to bypass the top-level file filters (i.e. those located above the FSNK engine). This function has many parameters and it is recommended to read about them in the documentation. To close the file, use the FsnkIoClose routine.

void
__stdcall
MyFsCallbackRoutine (
    IN PVOID pContext,        // Application-defined value.
    IN PFSNK_OP_DATA pData,        // General information of the request.
    IN PFSNK_OP_PARAMS pParams)    // Operation-specific information.
{
    ULONG uError = 0;
    HANDLE hFileHandle = NULL;
    FSNK_CREATE_RESULT uCreateResult = 0;
    FSNK_CREATE_OPTION uCreateOptions = 0;

    // Create options bitmask.
    uCreateOptions =
        FsnkOptionNonDirectory |        // Open if not a directory only.
        FsnkOptionSynchronousIoNonAlert |    // Open for synchronous access.
        FsnkOptionRandomAccess;            // Don't read ahead and cache.

    // Open the indicated file object.
    if (! FsnkIoCreate (
        &hFileHandle,        // File handle will be returned here.
        &uCreateResult,        // What was done will be returned here.
        pData -> FileName,    // Full name of the object to open.
        FILE_GENERIC_READ,    // Request read access only.
        FsnkDispositionOpen,    // Open existing file only.
        FsnkShareRead,        // Don't allow write access while reading.
        uCreateOptions,        // Options for create operation.
        FsnkIoBypassFilters))    // Don't generate recursive calls to us.
    {
        // Error handling goes here.
        uError = FsnkGetErrorCode ();
        ...
    }
    ...
    // Not needed anymore, close file.
    FsnkIoClose (&hFileHandle);
}

After opening the file, you can use the other functions of this group to work with it. For example, the FsnkIoRead function for reading data from an open file. Similarly to the other functions of this group, the reading request will bypass the Win32 and Native levels, and top-level filters will not view the request either. The input parameters of the function include the object handle, the buffer address for the data read from the file, buffer size in bytes and some flags for the operation control. At output, in case of success, the number of bytes actually read will be returned in *puBytesRead.

ULONG uError = 0;
ULONG uBytesRead = 0;
HANDLE hFileHandle = NULL;
BYTE auDataBuffer [1024] = {0};
...

// Read file header from the beginning.
if (! FsnkIoRead (
    hFileHandle,        // File handle with read access.
    auDataBuffer,        // Buffer for the data.
    sizeof (auDataBuffer),    // Size of the specified buffer, in bytes.
    0,            // Offset value to start reading from, in bytes.
    FsnkIoNoBuffering,    // Don't use caching for the particular operation.
    &uBytesRead))        // Number of bytes that was actually read.
{
    // Error handling goes here.
    uError = FsnkGetErrorCode ();
    ...
}

This group contains a considerable number of functions, and they are all described in details in the documentation. We recommend you to read the documentation carefully.

Callbacks and Structures

This section describes the callback function, which is called for each event in the file system, and the callback function for informing about events that occur in the engine. There is also a description of the structures that store information about intercepted file system events.

File system event callback

This callback routine informs an application about file system events. Event information is represented by the callback in two structures: FSNK_OP_DATA and FSNK_OP_PARAMS. Some additional information can be obtained via library API functions.

The pData parameter is the address of the structure that stores the information common for each operation: the operation code (creating, deleting, writing etc.), operation status, process ID of the requestor, SID of the user account used as security context for the operation, and a few more. The pParams input contains the address of the structure holding the information specific for a particular operation. For example, for reading or writing operation it is the number of bytes read or written from a file, and for creating a link it is the link file name.

Some values are unique only as long as they are used in the system. The example is the volume ID: it is unique until the volume is unmounted. You should have an up-to-date list of such identifiers if you want to use them in your application, and the callback will help you with this.

This routine is called in the context of one of the library threads. You should return control to the library as soon as possible, as the number of threads that process file system event notifications is limited. The library will call this routine only for a completed operation, so there is no possibility to block the operation or to change its parameters.

Engine event callback

This callback is optional, but it can be useful for informing about various events in the product engine. The callback parameters receive the following data: the application-defined context (for example, the address of a structure), library event code, and Win32 error code if applicable. The last parameter represents the specific details of the event. For instance, it may indicate an engine component where the failure occurred, which is useful for debugging.

This routine can be called in the context of an arbitrary thread. It is recommended for this callback to be always set, because in case of failure there will be a chance to reinitialise the library and continue working by returning ‘true’ from a callback. For more information, see the documentation.

General information about file system events

The FSNK_OP_DATA structure stores general information about file system events. By using it, you can get information about the operation type, process, session, and the user who requested the operation.

The structure stores the following data: the operation flags bitmask, operation code (read, write etc.), and the operation status. This is followed by the related identifiers: the unique identifiers of the thread, process and session, logon session ID, account SID used to execute the operation, the ID of the volume where the file object is located. And, finally, the NT-style name of the file system object and the offset to the relative path to the file in characters.

Please remember that the indicated thread, process and session ID are not unique over the time. For example, the thread identifier is valid only until the thread is terminated. The current FSNK version cannot track the creation and termination of threads and processes, but there are plans to add this functionality in future releases.

Specific information about file system events

Using the FSNK_OP_PARAMS structure, you can get the operation-specific information about many operations, such as creating, closing, reading and writing, mounting, changing a volume label, creating a links, moving or renaming, changing a security descriptor, volume expansion and shrinking, etc.

For instance, the following data will be available for open/create operations: the type of request, the requested access to the object, file attributes, the action that should be taken if the file already exists, the availability of the file to other applications, the bitmask of the parameters defining various aspects of the execution of create operation, the initial size of a new file, and the final results of the operation (created, replaced, opened, etc.).

Version History

2014.04.10 v1.0

• Initial release of the File System Notification Kit. Your feedback is very welcome.