XFile - Extending the Win32 File API for Server Applications

• Download demo project - 127 Kb

Introduction

CXFile represents a collection of file-handling code snippets I have put together over a few years working on NT client/server systems. My main reason for writing these functions was to make it easier to implement such systems - I wanted consistent interfaces, plenty of diagnostics about what was going on, and extensions that go beyond anything in the Win32 API. I also wanted to make it independent of MFC, because some older systems I work on use only Win32 SDK.

CXFile Features

When I started pulling together different pieces of code for CXFile, one of the first things I realized was that I would have to do something about all the TRACE statements that were scattered throughout the code. I am a big fan of using TRACE statements as a way to monitor actual system operation, but I was afraid I would have to yank all TRACE statements out because I did not want to be tied to MFC. I was very reluctant to do this, because alternative is to put debug output statements in application code, which becomes very messy.

Luckily I found an article by Paul Mclachlan, "Getting around the need for a vararg #define just to automatically use __FILE__ and __LINE__ in a TRACE macro". Paul's TRACE replacement is not only free of MFC dependency, it also provides file and line number output, is thread-safe, and is completely encapsulated in one header file. I have made some minor changes to Paul's class (such as adding thread ID to the output), and so I renamed it to XTrace.h to prevent any conflicts.

With the big TRACE problem out of the way, I could now decide how to use TRACE in a consistent manner. I decided I wanted diagnostics of two types: a basic informational-type debug message, that reports values and code flow; and an error diagnostic for API failures and other serious problems. So I use the standard TRACE macro for debug messages, and a new TRACEERROR macro for API failures. It is easy to disable one or both of these macros, by replacing them with usual

#define TRACE ((void)0)

definition, and I have included this capability in XFile.cpp source.

Now for CXFile features. In addition to basic file operations such as opening/creating, reading, writing, copying, deleting, and renaming files, CXFile includes extended functions:

• Compare() - Compare two files to see if they are same (binary compare).

  ////////////////////////////////////////////////////////////////////////////
  //
  // Compare()
  //
  // Purpose:     Binary compare two files
  //
  // Parameters:  lpszFile1 - file 1 name
  //              lpszFile2 - file 2 name
  //              pbResult  - pointer to BOOL that receives result of compare;
  //                          TRUE = files are identical.  pbResult is valid
  //                          only if Compare() returns TRUE.
  //
  // Returns:     BOOL - TRUE = no problems encountered during compare, see
  //                     pbResult for result of compare.  If FALSE returned,
  //                     pbResult is meaningless.
  //

• Rollover() - Rollover a file. This is very common to do at midnight (or last day of week/month), by renaming file to a new name, and starting again with empty file. This function also allows you to specify whether you want to add an entry to a rollover log file, and whether you want to zip the rollover file.

  /////////////////////////////////////////////////////////////////////////////
  //
  // Rollover()
  //
  // Purpose:     Rollover file to new file and optionally zip, continue
  //              with empty file
  //
  // Parameters:  lpszRolloverFileName     - name to use for new rollover file
  //              lpszFileNameBuffer       - buffer for rollover file name;  if
  //                                         not NULL, the rollover file name
  //                                         will be copied into this buffer
  //              dwFileNameBufferSize     - size of lpszFileNameBuffer in TCHARs;
  //                                         if 0, rollover file name will not be
  //                                         copied into lpszFileNameBuffer
  //              bFailIfExists            - operation if file exists:
  //                                         TRUE = if new file already exists,
  //                                         the function fails
  //                                         FALSE = if new file already exists,
  //                                         the function overwrites the existing
  //                                         file and succeeds
  //              bZip                     - operation if Rollover() succeeds:
  //                                         TRUE = zip rollover file; name of
  //                                         archive will be same as rollover file,
  //                                         with .zip extension; then delete
  //                                         rollover file
  //                                         FALSE - do not zip
  //              bGenerateRolloverLogFile - create entry in rollover log file:
  //                                         TRUE = create entry
  //                                         FALSE = do not create entry
  //
  // Returns:     BOOL - TRUE = success
  // 

• Shrink() - Shrink file by discarding data from beginning of file. After file has been shrunk, the Shrink function will optionally look for the start of next record/line, and further reduce size to eliminate partial record.

  ///////////////////////////////////////////////////////////////////////////////
  //
  // Shrink()
  //
  // Purpose:     Shrink an open file when it exceeds a size limit
  //
  // Parameters:  dwMaxFileSize   - maximum file size in bytes
  //              dwShrinkToSize  - size in bytes to shrink file to
  //              lpDelimiter     - byte array containing one or more
  //                                characters that comprise line/record ending -
  //                                e.g., "\r\n";  use NULL if partial record
  //                                elimination is not desired
  //              dwDelimiterSize - size in bytes of delimiter array
  //
  // Returns:     DWORD - Number of bytes file was shrunk.  0 = file was not
  //                      shrunk, (DWORD)-1 = error occurred.
  //
  // Notes:       Shrink may be called anytime after a file has been opened.
  //              Shrink will have no effect on files opened as readonly.  If
  //              the file size exceeds dwMaxFileSize, it will be shrunk to
  //              the size specified by dwShrinkToSize.  Regardless of the type
  //              of file (text or binary), the first part of the file will be
  //              discarded, leaving only the last dwShrinkToSize bytes.
  //
  //              Then, if lpDelimiter is non-NULL, this byte array will be
  //              searched for in the first 64K bytes of the new file.  If found,
  //              the new file will begin at the first byte past the lpDelimiter
  //              array that was found.  This ensures that text files and other
  //              record-oriented files will begin at the beginning of a line
  //              or record.  Note that this may cause the new file to be smaller
  //              than the specified dwShrinkToSize.
  //
  //              lpDelimiter is treated as a byte array by Shrink, whether the
  //              app is compiled for Unicode or not.  For example, if you pass
  //              _T("\r\n") as the delimiter array. and 2*sizeof(TCHAR) as the
  //              delimiter size, Shrink will look for the sequential bytes
  //                       0x0D 0x00 0x0A 0x00
  //              in the file if your app is compiled for Unicode.  If it is
  //              compiled for ANSI, Shrink will look for the sequential bytes
  //                       0x0D 0x0A.
  //              So it is up to you to use a delimiter string that matches what
  //              is actually in the file, regardless of how your app is compiled.
  //              Shrink does no conversion on the data it reads from the file.
  //
  //              Shrink will be useful to keep frequently appended-to files from
  //              growing too large.  You should only use Shrink on files where
  //              the most recent data is appended to the end of the file - for
  //              example, log files that are written by server processes.
  //
  //              Shrink() will always attempt to position the file at EOF before
  //              returning.
  // 

• Zip() - Compress file and create zip archive. This is frequent requirement in production environments. For example, enterprise XML files can become very large, and will easily fill a server's hard drive in a short time if not managed. When compressed with Zip, this issue goes away. The zip engine is implemented in single .cpp/.h pair of files - no lib or DLL is necessary. Currently CXFile::Zip() is limited to creating an archive containing only one entry, although zip engine is capable of producing multi-file archives. The zip archive that is produced can be read by other compression programs such as WinZip. If you do not need zip capability, you can define DO_NOT_INCLUDE_XZIP at top of XFile.cpp, and remove XZip.cpp and XZip.h from your project.

  ///////////////////////////////////////////////////////////////////////////
  //
  // Zip()
  //
  // Purpose:     Zip a file
  //
  // Parameters:  lpszZipArchive - name of zip archive to create
  //              lpszSrcFile    - name of file to zip; the file name and
  //                               extension from lpszSrcFile will be used as
  //                               the entry name in the zip file
  //              bFailIfExists  - operation if zip archive exists:
  //                               TRUE = if zip archive already exists, the
  //                               function fails
  //                               FALSE = if zip archive already exists, the
  //                               function overwrites the existing zip archive
  //                               and succeeds
  //              bDeleteSrcFile - operation if zip creation succeeds:
  //                               TRUE = delete lpszSrcFile
  //                               FALSE = do not delete lpszSrcFile
  //
  // Returns:     BOOL - TRUE = success
  //
  // Notes:       Zip() compresses one file and stores it in a new zip archive.
  //              It does not handle more than one file per archive.
  //
  //              Zip() supports Unicode in the following way:  the names of 
  //              the zip archive and zip source file may be passed as Unicode
  //              strings (if built with _UNICODE defined).  However, the source
  //              file name is stored internally in the zip archive as ANSI. 
  //              This is a basic limitation of the zip engine.
  // 

• Printf() - Write formatted output to a file using printf formatting.

  ///////////////////////////////////////////////////////////////////////////
  //
  // Printf()
  //
  // Purpose:     Write printf-formatted output to an open file
  //
  // Parameters:  lpszFmt - format specification (see printf)
  //              ...     - variable number of args
  //
  // Returns:     int - number of characters actually written, or negative
  //                    value if error occurs
  //
  // Notes:       A maximum of MAX_PRINTF_CHARACTERS characters can be
  //              outputted.
  // 

• Limit File Size - The file size can be limited automatically as data is written, either by calling Shrink to discard from the beginning, or by discarding data from end of the file.

  ///////////////////////////////////////////////////////////////////////////////
  //
  // SetSizeLimit()
  //
  // Purpose:     Set parameters to limit file size
  //
  // Parameters:  dwSizeLimitBytes - maximum file size in bytes;  if the limit
  //                                 type is SHRINK, when dwSizeLimitBytes is
  //                                 exceeded, the "shrink to" size will be set at
  //                                 dwSizeLimitBytes / 2.
  //              eType            - limit type:
  //                                 TRUNCATE - file size will be limited by
  //                                 discarding writes when the file has reached
  //                                 dwSizeLimitBytes
  //                                 SHRINK - file size will be limited by
  //                                 discarding first half of file, and then
  //                                 using lpDelimiter string to discard partial
  //                                 record.  See Shrink() for more details.
  //              lpDelimiter      - byte array containing one or more
  //                                 characters that comprise line/record ending -
  //                                 e.g., "\r\n"
  //              dwDelimiterSize  - size in bytes of delimiter array
  //
  // Returns:     BOOL - TRUE = success
  //
  // Notes:       Setting a file size limit should only be done for files that
  //              are being written sequentially - i.e., new data is written to
  //              the end of the file.
  //
  //              The file size limit will be enforced when Flush() is called,
  //              which typically happens when the internal CXFile write buffer
  //              is full.  The first time Flush() is called, the data will always
  //              be written to the file, since the file pointer is at the
  //              beginning of the file and the max size hasn't been exceeded yet.
  //              Therefore, the effect of the max size limit will only be seen
  //              when it is larger than the internal buffer size, so that
  //              multiple Flush()'s will occur, and the max size limit will be
  //              triggered.
  //

• Mapped reads - read a file using memory-mapping. After calling CXFile::OpenMapped(), you use Read() API for reading a mapped file - this is same one used to read a disk file. There is no need to use MapViewOfFile or UnmapViewOfFile. One important thing to note: with mapped reads, the file offset (and therefore the read buffer size) must be a multiple of the system's virtual memory allocation granularity, or the next call to MapViewOfFile() will fail. Use CXFile::GetAllocationGranularity() to retrieve this value and allocate a buffer.

  ///////////////////////////////////////////////////////////////////////////////
  //
  // OpenMapped()
  //
  // Purpose:     Open file for mapped reads
  //
  // Parameters:  lpszFile  - file name
  //
  // Returns:     BOOL - TRUE = success
  // 

• Unicode header (Byte Order Mark, or BOM) - You can optionally request that a BOM is written to the beginning of a file. According to Unicode Consortium (see UTF & BOM), the signature at the beginning of certain data streams (such as unmarked plaintext files). is referred to as the BOM character, for Byte Order Mark. For little-endian systems such as those based on Intel x86 processors, the BOM is encoded as the two byte sequence FF FE (hex values). For big-endian systems, the BOM is FE FF. Where a BOM is useful is with files that are typed as text, but not known to be in either big- or little-endian format. In that situation, BOM serves both as a hint that the text is Unicode, and also what type of byte-ordering was used to create the file.

  You have option to specify Unicode header in Open():

  ///////////////////////////////////////////////////////////////////////////////
  //
  // Open()
  //
  // Purpose:     Open file
  //
  // Parameters:  lpszFile  - file name
  //              bReadOnly - TRUE  = file will be opened for read only
  //                          FALSE = open file for reading and writing
  //              bTruncate - TRUE  = open existing and truncate, or create new
  //                          FALSE = open existing (do not truncate)
  //                                  or create new
  //              bUnicode  - TRUE  = file is Unicode text file, write Unicode
  //                                  header (Byte Order Mark, or BOM)
  //                          FALSE = do not write Unicode header
  //
  // Returns:     BOOL - TRUE = success
  //
  // Notes:       Some text editors (e.g., TextPad) and other applications write
  //              a Unicode header of two bytes (0xFF followed by 0xFE).  However,
  //              TextPad seems to have no problem figuring out that the file
  //              contains UTF-16 even if the header is not present.  You should
  //              test this with your application to see if it makes any
  //              difference.  In particular, you should verify that the presence
  //              of a BOM does not interfere with your app's read operations.
  // 

• GetRecord() - read byte data from an open file, until the buffer is full or the record delimiter is found.

  ///////////////////////////////////////////////////////////////////////////////
  //
  // GetRecord()
  //
  // Purpose:     Read byte data from an open file.  The file may have been
  //              opened either with Open() or OpenMapped(). Up to dwBytes of
  //              data are read into buffer specified by lpszBuffer, or until
  //              the record delimiter is found.  See SetRecordDelimiter().
  //              Each successive GetRecord() will return the next record,
  //              until 0 is returned at the end of file.
  //
  // Parameters:  lpszBuffer - pointer to byte buffer that byte data will be
  //                           read into
  //              dwBytes    - size of buffer in bytes
  //
  // Returns:     DWORD - number of bytes read, or (DWORD)-1 if error; EOF found
  //                      if (DWORD)-2 returned.
  //
  // Notes:       GetRecord() starts reading from current file position.  Use
  //              of Seek() will interfere with the sequential reading of data.
  //
  //              GetRecord() may be used on both text and binary files.  The
  //              record delimiter (specified by SetRecordDelimiter()) will be
  //              searched for as a sequence of bytes, regardless of whether
  //              the file is ANSI text, Unicode, or binary.
  // 

• Prepend() - write data to the beginning of an open file.

  ///////////////////////////////////////////////////////////////////////////////
  //
  // Prepend()
  //
  // Purpose:     Write byte data to the beginning of an open file.  Data is
  //              written to beginning of file, before the data that was in the
  //              file.  The data that was in the file is preserved and begins
  //              immediately after the new data.
  //
  // Parameters:  lpszBuffer - pointer to byte buffer containing data
  //              dwBytes    - number of bytes in buffer
  //
  // Returns:     DWORD - number of bytes written, or (DWORD)-1 if error
  // 

• Search() - search an open file for a byte sequence.

  ///////////////////////////////////////////////////////////////////////////////
  //
  // Search()
  //
  // Purpose:     Search for a sequence of bytes
  //
  // Parameters:  lpSearch         - byte array containing one or more values
  //                                 that comprise the search string;  any value
  //                                 permitted, including nul.
  //              dwSearchSize     - size in bytes of search string
  //              bCaseInsensitive - TRUE = search will be done ignoring case
  //                                 (of those characters that fall in the ANSI
  //                                 range a-z and A-Z);
  //                                 FALSE - characters in the search string and
  //                                 characters in the file will be compared
  //                                 "as is", with no case conversion.
  //
  // Returns:     DWORD - >= 0 if lpSearch string was found; return value is the
  //                      file position of the string found.  If the search failed,
  //                      (DWORD)-1 is returned.
  //
  // Notes:       After a successful search, the file position will be set to the
  //              first character position of the string found.  After an
  //              unsuccessful search, the file position will be unchanged.
  //
  //              The search always starts at (or one byte past) the current file
  //              position;  Seek() is not called prior to starting the search.
  //              If the starting file position is past the beginning of the file
  //              (i.e., not 0), then the search will start at one past the current
  //              file position.  Otherwise, it will start at the current file
  //              position (the beginning of the file).  This allows successive
  //              searches to be done without having to move the file pointer past
  //              a previous successful search.
  //
  //              Searches do not wrap.
  // 

• Write buffering - CXFile buffers file writes to its own buffer, and writes buffer to file when it becomes full. This reduces file API calls, especially for short data records. The buffer size can be set programmatically.
  
  
• Read/write statistics - The number of reads/writes, as well as total number of bytes read/written, is available.
  
  
• Two-level non-MFC TRACE output - Each Win32 file API is checked for its return code, and if it failed, GetLastError() code is converted to string format and dumped. All errors are reported by TRACEERROR(), while other useful debug information is reported by TRACE(). With Paul Mclachlan's excellent TRACE class, both types of output include source module and line number. One or both types of output can also be easily disabled within XFile.cpp.
  

Data Conversions

Writing Data

When writing files, CXFile will only write whatever data is in byte buffer that is passed to it, with one exception: if CXFile object has been constructed with the Unicode flag set to TRUE, or has been opened with the Unicode flag set to TRUE, then CXFile will write Unicode header (see above) when writing data at beginning of file. This flag should be used only for plain text files.

Reading Data

When reading files, CXFile will read whatever data is in file, and return it in buffer passed to it. No CR/LF or any other type of conversion is done.

Zip Files

The zip engine used in CXFile does not care what type of data is in files that you zip. However, the names of the files being zipped are handled as ANSI internally by zip engine. If you look at XZip.cpp, the ZipAdd function is prototyped as

ZRESULT ZipAdd(HZIP hz, const TCHAR *dstzn, void *src, unsigned int len,
               DWORD flags)

Internally, dstzn parameter is converted from Unicode to ANSI, and this ANSI string is used as the entry name within the zip archive. The src parameter is name of source file (that will be compressed by ZipAdd), and should also be passed as a Unicode string if _UNICODE is defined.

How To Use

To integrate CXFile class into your app, you first need to add following files to your project:

• XFile.cpp
• XFile.h
• XZip.cpp
• XZip.h
• XTrace.h

The XTrace.h file is optional, and can be excluded. If you exclude it, comment out #include "XTrace.h" line from XFile.cpp, and uncomment #define TRACEERROR ((void)0) line. The XZip.cpp and XZip.h files are also optional, if you do not need Zip() function. Uncomment line #define DO_NOT_INCLUDE_XZIP at top of XFile.cpp.

If you include CXFile in project that uses precompiled headers, you must change C/C++ Precompiled Headers settings to Not using precompiled headers for XFile.cpp and XZip.cpp.

Next, include the header file XFile.h in appropriate project files (stdafx.h usually works well). Now you are ready to start using CXFile. There are many notes concerning usage of various functions in XFile.cpp. Please read all function header for each function you wish to use.

There are two ways to construct CXFile object. Typical way is to specify file name in ctor:

CXfile file("myfile.txt");
file.Printf("This is test %d", nTest);

If you do not specify file name in ctor, you must call Open():

CXfile file;
file.Open("myfile.txt");
file.Printf("This is test %d", nTest);

Known Limitations

// Known limitations:
//     1.  File size must be smaller than a DWORD value.
//     2.  Maximum number of Printf() output characters (TCHARs) must be less
//         than MAX_PRINTF_CHARACTERS.
//     3.  When using the mapped file functions, the read buffer MUST be a
//         multiple of system's virtual memory allocation granularity, or
//         the next call to ReadMapped will fail because the file offset
//         will be illegal.  (The first call will succeed because the file
//         offset will be 0).
//     4.  XFile has only been tested with files. 

Demo App

The XFileTest.exe demo tests the APIs in CXFile. Here is output from Test 1, that shows a binary file created and then read. Its contents are then checked.

Frequently Asked Questions

1. Why use CXFile at all? Why not just use the Win32 file APIs?
   Aside from the extended functions like Zip(), Shrink(), and Rollover(), the main reason is to have better error reporting. Instead of having to check the return code, and then do error reporting yourself after each API call, all you have to do is check the return code - CXFile does the error reporting for you.
   
   
2. I don't want to run my app under the debugger all the time, just to get the TRACE output. How can I see all this error reporting?
   You can use the excellent free utility DebugView from Sysinternals. This allows you to see all TRACE output from your debug builds. One very nice feature of DebugView that I cannot live without is the ability to filter the output, and colorize any line that contains a particular string. For example, you can set the filter to color any line containing "error" with red background and white text. You can probably guess what my filters are from this screenshot:

   
   

3. Can I use XFile in non-MFC apps?
   Yes. It has been implemented to compile with any Win32 program.
   
   
4. When I try to include XFile.cpp in my MFC project, I get the compiler error XFile.cpp(2611) : fatal error C1010: unexpected end of file while looking for precompiled header directive. How can I fix this?
   When using XFile in project that uses precompiled headers, you must change C/C++ Precompiled Headers settings to Not using precompiled headers for XFile.cpp and XZip.cpp. Be sure to do this for All Configurations.

   
   

5. When I try to build the demo app, I get the linker error LINK : fatal error LNK1104: cannot open file "mfc42u.lib" Error executing link.exe. How can I fix this?
   The default installation options of Visual C++ v6.0 don't install the Unicode libraries of MFC, so you might get an error that mfc42u.lib or mfc42ud.lib cannot be found. You can fix this either by installing the Unicode libs from the VC++ install CD, or by going to Build | Set Active Configuration and selecting one of the non-Unicode configurations.

   

   You can configure the Visual Studio toolbars to include the Select Active Configuration combobox. This lets you see at a glance what configuration you are working with.
   

6. I am trying to use mapped reads, but I get a debug assertion in XFile.cpp. What's wrong?
   Check the TRACE output. If you see a line that says WARNING: file offset is not a multiple of system's virtual memory, then you are trying to use an invalid file offset. With mapped reads, the file offset (and therefore the read buffer size) must be a multiple of the system's virtual memory allocation granularity, or the next call to MapViewOfFile() will fail. Here is what MSDN says about this:

   The combination of the high and low offsets must specify an offset within the file that matches the system's memory allocation granularity, or [MapViewOfFile] fails. That is, the offset must be a multiple of the allocation granularity. Use the GetSystemInfo function, which fills in the members of a SYSTEM_INFO structure, to obtain the system's memory allocation granularity.

   You can use CXFile::GetAllocationGranularity() to retrieve the system's memory allocation granularity and allocate a buffer. Search for "granularity" in XFile.cpp for more information.
   

7. I don't need the Zip function. Can I exclude XZip.cpp?
   Yes. Uncomment the following line at the top of XFile.cpp:

   //#define DO_NOT_INCLUDE_XZIP

8. Can we use XFile and XZip in our (shareware/commercial) app?
   Yes, you can use XFile without charge or license fee. It would be nice to acknowledge my Copyright in your About box or splash screen, but this is up to you. XZip has its own licensing requirements. Basically, it is free to use any way you want, but you must acknowledge use in your documentation. See header of XZip.cpp for details.
   
   
9. Does XFile handle pipes? mailslots? other types of devices?
   XFile has not been tested with anything other than files.

Acknowledgments

• The code in XZip.cpp and XZip.h is included with kind permission of Lucian Wischik. It is based on code from Info-ZIP. Please see XZip.cpp for Info-ZIP's copyright and restrictions.
  
  
• The XTrace code is based on an article by Paul Mclachlan, "Getting around the need for a vararg #define just to automatically use __FILE__ and __LINE__ in a TRACE macro".
  
  
• The CXListBox class is taken from my own article, XListBox - Owner-draw CListBox with selectable text and background colors.

Revision History

Version 1.2 - 2003 May 15

• Enabled use on Win9x - removed dependency on MoveFileEx.
• Added Prepend function.
• Added GetRecord function.
• Added GetTotalRecordsRead function.
• Added Search function
• Added ClearBuffer function.
• Fixed SetSizeLimit and Shrink parameters to accept byte array rather than string, and also added parameter for byte array size. This allows sequence of non-text values to be used for delimiter.
• Added FILE_FLAG_SEQUENTIAL_SCAN flag to CreateFile, suggested by mattb79.
• Added #define XFILE_ERROR ((DWORD)-1)
• Added #define XFILE_EOF ((DWORD)-2)

Version 1.1 - 2003 May 5

• Initial public release

Usage

This software is released into the public domain. You are free to use it in any way you like. If you modify it or extend it, please to consider posting new code here for everyone to share. This software is provided "as is" with no expressed or implied warranty. I accept no liability for any damage or loss of business that this software may cause.