Click here to Skip to main content
Click here to Skip to main content
Go to top

tailFile: Blazing fast tail utility (watch files in real time)

, 9 Dec 2013
Rate this:
Please Sign up or sign in to vote.
Watch files in real time with this cmd-line utility.

Introduction  

I searched  CodeProject for a good tail utility and noticed there are none implemented in C# and the latest one which has been developed was back in 2006.  I needed a good log tailing utility so I wrote this Console App in C#.

Mine has some additional features:

  1. Can tail a 5 Gigabyte (or more) file for just the _last_ N bytes you want to see.  Yes, I've tested this on huge files and it is blazing fast, because it opens the file then moves the file pointer backwards to the point in the file you want to see.  It's instant on huge files  (More about this below.) 
  2. Allows you to continually tail any file.  This was the option I really needed.  I wanted to be able to run the utility in a cmd window where it would sit idly and then update any time data was written to my log file. Works great and doesn't eat your processor.  
  3. Can be used along with other cmd line utilities such as _find_.  If you want to only see lines which have certain strings you can pipe tailFile's output through find and only see those lines (more below).
  4. Notepad++ / Admin / Windows 7 :
    Normally I use Notepad++ to watch files, but recently my work machine was upgraded to Windows 7 and I don't have full admin rights so Notepad++* (currently version 6.3) would tail the file one time and then stop.  It bugged me into writing this utility. 

* This  behavior where it fails to tail the file was supposedly fixed in ver. 6.1.6, but the problem still occurs for me.

Background   

I am developing a windows svc.  The service writes test info and errors to a log file since there is no UI.  I needed a good command line utility which would allow me to see what the svc is writing to my log file so I wrote this program.  

ALERT!!! Bug Fix  - Release 1.3.1.0 

1.3.1.0 on 4/30/2013 ### BUG FIX #### Since I am now reading the BOM characters out of the target file, I forgot to include the open for share indicator when opening the FileReader.  That means the app. fails when attempting to open a file that is already in use by another file.  Fail!  I fixed that in version 1.3.1.0 I apologize for any inconvenience.   

Without this fix, if you run this against a file that is being written to by another process (the entire point of tailFile) then you will get an error that you cannot open the file because it is already in use. 

The code fix is as simple as adding one parameter on one line of the code where I open the target file to read the BOM.  Please note that last parameter on the following line

 reader = new FileStream(inArgs[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite); 

Updated To Support Alternate File Formats 

 I've updated the tailFile utility to support three new file formats: 

  1. Unicode (UTF16)  
  2. UTF-8
  3. Big Endian (UTF) 

Attempt To Determine File Format 

The code now attempts to automatically determine which format the target file (file which will be tailed) is in.

It does so by checking for a BOM (Byte Order Mark): two bytes at the beginning of the file which indicate the file format.

You can see where my code now attempts this in the method : DetermineFileType()

In that method after I open the file I examine the first two bytes.  The code looks like:

//Check for BOM for unicode file
   if (readBuffer[0] == 255)
    {
       if (readBuffer[1] == 254)
       {
           return filetype.Unicode;
       }
   }
// check for BOM for UTF8
   if (readBuffer[0] == 239)
   {
      if (readBuffer[1] == 187)
      {
          return filetype.UTF8;
      }
   }   
// check for BOM for BigEndian
   if (readBuffer[0] == 254)
   {
     if (readBuffer[1] == 255)
     {
        return filetype.Big;
     }
   }
   return filetype.ASCII; 

You can see that each file format BOM is defined by two chars and it was easy to do this check, even though it is a bit of a brute-force method to do so.   If none of the two-byte BOMs are found, then I simply drop back to ASCII mode.   

If you examine the entire listing for the method, you'll also see if any exception is thrown then I drop back to ASCII mode.   

Why Am I Determining File Format?

The entire reason I determine the file format is because it was reported that my tailFile utility didn't work with Unicode format.  It did work, however, the output looked incorrect.  When you examine the image below, you'll see that upon output, tailFile previously would print the actual char (ASCII) and then print the empty byte as a char (white-space) also.  

Unicode: The Problem Shown In the Image No Longer Occurs   

unicode format file 

Conversion of Bytes Is Only For Console Output : Little Impact 

Once I determine the file format I simply convert the bytes for output to the Console.  Since this is just for output there is very little impact and even if it fails to determine the file format tailFile will still continue to output the bytes.

Brute-Force Method For Determining File Format Doesn't Always Work

While testing and researching I found that the brute-force method for determining the file format doesn't always work.  Why not?  It is reported that some files which may be in Unicode, UTF8 or Big Endian format may not contain the BOM.  If they do not then tailFile will fall back to ASCII.

 In an effort to resolve this issue I now allow the user to force tailFile to convert the output to his/her choice simply by adding a string representing a file format on the command line.

This allows the user to force tailFile into a specific file format, even if the target file does not contain the BOM.

The following are the strings that can be used on the command line (examples of full command line below).

Unicode = uni 
UTF8 = utf 
Big Endian = big   

New Version Still Works With Original Arguments 

However, I have insured that the changes to the arguments still allow all command lines which worked in the previous version (1.2.0.0) will still work. 

Using the tailFile Utility    

Here are some sample command lines to show you how to run the utility. Usage: tailFile <filename> <numberOfBytesToDisplay> [filetype] [-c]ontinue tailing 

  • tailFile c:\test\mydata.dat 500 // displays the last 500 bytes of the file and exits 
  • tailFile c:\test\mydata.dat 500 utf  // forces the format to UTF8 mode - displays the last 500 bytes of the file and exits 
  • tailFile c:\test\mydata.dat 500 uni // forces the format to Unicode mode - displays the last 500 bytes of the file and exits   
  • tailFile c:\test\mydata.dat 500 big // forces the format to Big Endian mode - displays the last 500 bytes of the file and exits  
  • tailFile c:\test\mydata.dat 500 uni -c // forces Unicode mode displays the last 500 bytes of the file and continues to tail 
  • tailFile c:\test\mydata.dat 500 -c // displays the last 500 bytes of the file and continues to tail
  • tailFile c:\test\mydata.dat 500 -c | find /I "test"  // display only lines where "test" is found, continue

Note: The last two arguments denoted in the [ ] (square brackets) are optional. You can have one or the other, neither or both.  However, if you have both, the [filetype] must come first.
 

Starting the Utility Without Any Arguments Provides Sample Command-Lines 

 

 To stop the program when continually tracing do one of the following: 

  • CTRL-C 
  • q <ENTER>

Entire Program, Only 158 Lines

The entire program is only 158 lines, so I'll reproduce it below, but first I'll provide some explanation about how it works and code snippets as examples as I explain. 

Basically, it implements a timer which checks the size of the file.  If the file has not changed you'll see nothing.  However, if the file size has changed it will display those last bytes which were written (since last checking).

Code Explanation

Starting The Program

The code will only run if it is supplied with 2 or 3 arguments.  If you supply less or more you will see an error message and a usage statement showing you how to use the program.

That code is handled in the main program loop as a switch statement.

Providing Two Arguments Runs DisplayBytes()

The DisplayBytes() method does the following:

1. grabs the number of bytes you want to read from the arguments
2. Opens a filestream (name of file provided in args) using a filemode (bold in snippet below)  which allows other processes to still access (read/write) the file while tailFile is accessing the file.  This is very important because we only want to read bytes for display and we do not want to interrupt other processes which are writing to the file. 
 fs = new FileStream(args[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite); 

3. After we open the file then we want to make sure that we are not attempting to read more bytes (args value) from the file than actually exist in the file, so I do that with the one line of code shown in the snippet below.  This line uses the ternary operator to do a comparison of numberOfBytesToDisplay to the file.Length. If number of bytes to display is less then it uses that value, if it is equal to or greater than the size of the file then it uses the file length to determine the number of bytes to display.

numberOfBytesToDisplay = numberOfBytesToDisplay < fs.Length ? numberOfBytesToDisplay : fs.Length;

4. Next the program seeks backwards (notice the -numberOfBytesToDisplay passed in to Seek() ) through the file to the place where the first byte we want to read is and then reads the bytes into a buffer.  

fs.Seek(-numberOfBytesToDisplay, SeekOrigin.End); 
  fs.Read(buf, 0, (int)numberOfBytesToDisplay); 

All of the code is wrapped in a try...catch in case any exception is thrown it will display an error and call ExitProc().  ExitProc() displays a message and makes sure the main wait loop (Do...While) is exited by killing the process.

Why ExitProc ()? 

ExitProc method is necessary because this is a Console App and I use the ReadLine() method to cause the process to continue.  ExitProc() gives me a way to provide an error message to the user before closing and then to kill the process using the System.Diagnostics library (MS C# core).

Continuous Tail, Timer and UpdateDisplayBytes()

When you provide a third argument to the program of -c then the program will start a timer which runs every 500ms.  I have tested this and because it only checks to see if there is a change to the file it does not cause the program to eat your processor. 

Timer Elapsed Method

When the timer elapses it checks to see if the file has changed with the following line of

 if (checkedSize > currentFileSize) 

If the file size has not changed, then nothing is done.  It's very efficient so your processor isn't eaten up, even though there is a timer running.

Finally, if the file has changed, the DisplayUpdateBytes() simply checks the number of bytes the file has increased since it last checked and then it writes those bytes to the console.  Simple as that.

Try it out, I think it works well and it's helping me watch my Windows service. 

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Security.AccessControl;

namespace tailFile
{
    class Program
    {
        enum filetype { None, UTF8, Unicode,ASCII,Big };
        private static System.Timers.Timer fileWatcher;
        private static string[] inArgs;
        private static Int64 currentFileSize;
        private static bool firstTime = true;
        private static string stopString;
        private static filetype currentFileType;
        
        static void Main(string[] args)
        {
            // Capture the args for use with timer
            inArgs = args;

            switch (args.Length)
            {
                case  2:
                {
                    DisplayBytes(args);
                    break;
                }
                case 3:
                {
                    UserArgSetsFileType();

                    if (currentFileType != filetype.None)
                    {
                        // this indicates that the 3rd arg was a filetype from
                        // the user so they just want to view a number of bytes
                        // and exit, but they were trying to force a filetype
                        DisplayBytes(args);
                    }
                    else
                    {
                        if (args[2].ToLower() == "-c")
                        {
                            DisplayBytes(args);
                            fileWatcher = new System.Timers.Timer();
                            fileWatcher.Elapsed += new System.Timers.ElapsedEventHandler(fileWatcher_Elapsed);
                            fileWatcher.Interval = 500;
                            fileWatcher.Start();
                            WaitForMessages();
                        }
                        else
                        {
                            Console.WriteLine("Argument 3 is incorrect. Please check the [filetype] or try -c");
                            ExitProc();
                        }
                    }
                    break;
                }
                case 4:
                {
                    UserArgSetsFileType();
                    if (args[3].ToLower() == "-c")
                    {
                        DisplayBytes(args);
                        fileWatcher = new System.Timers.Timer();
                        fileWatcher.Elapsed += new System.Timers.ElapsedEventHandler(fileWatcher_Elapsed);
                        fileWatcher.Interval = 500;
                        fileWatcher.Start();
                        WaitForMessages();
                    }
                    else
                    {
                        Console.WriteLine("Argument 3 is incorrect. Please check the [filetype] try -c");
                        ExitProc();
                    }
                    break;
                }
                default:
                {
                    Console.WriteLine("Incorrect number of arguments supplied.");
                    ExitProc();
                    break;
                }
            }
        }

        static void UserArgSetsFileType()
        {
            // if the last arg is uni or utf, 
            // allow user to force the program into a specific mode
            if (inArgs[inArgs.Length - 1].ToLower() == "uni")
            {
                currentFileType = filetype.Unicode;
            }
            else if (inArgs[inArgs.Length - 1].ToLower() == "utf")
            {
                currentFileType = filetype.UTF8;
            }
            else if (inArgs[inArgs.Length - 1].ToLower() == "big")
            {
                currentFileType = filetype.Big;
            }
        }

        static filetype DetermineFileType()
        {
            // now supports UNICODE, UTF8, Big Endian and ASCII
            FileStream reader = null;
            try
            {
                reader = new FileStream(inArgs[0], FileMode.Open, FileAccess.Read);
                byte [] readBuffer = new byte[2];
                reader.Read(readBuffer, 0, 2);
            
                //Check for BOM for unicode file
                if (readBuffer[0] == 255)
                {
                    if (readBuffer[1] == 254)
                    {
                        return filetype.Unicode;
                    }
                }
                // check for BOM for UTF8
                if (readBuffer[0] == 239)
                {
                    if (readBuffer[1] == 187)
                    {
                        return filetype.UTF8;
                    }
                }
                // check for BOM for BigEndian
                if (readBuffer[0] == 254)
                {
                    if (readBuffer[1] == 255)
                    {
                        return filetype.Big;
                    }
                }
                return filetype.ASCII;
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                ExitProc();
                return filetype.ASCII;
            }
            finally
            {
                reader.Close();
            }
        }

        static void WaitForMessages()
        {
            do
            {
                stopString = Console.ReadLine();
            }
            while (stopString.ToLower() != "q");
        }

        public static void ExitProc()
        {
            Console.WriteLine("Usage: tailFile <filename> <numberOfBytesToDisplay> [-c]ontinue tailing  [filetype]");
             Console.WriteLine("Usage: tailFile <filename> <numberOfBytesToDisplay> [-c]ontinue tailing  [filetype]");
            Console.WriteLine(@"Ex.1: tailFile c:\test\mydata.dat 500");
            Console.WriteLine(@"Ex.1: tailFile c:\test\mydata.dat 500 utf8");
            Console.WriteLine(@"Ex.2: tailFile c:\test\mydata.dat 500 big");
            Console.WriteLine(@"Ex.2: tailFile c:\test\mydata.dat 500 -c");
            Console.WriteLine(@"Ex.2: tailFile c:\test\mydata.dat 500 utf8 -c");
            Console.WriteLine(@"Ex.2: tailFile c:\test\mydata.dat 500 uni -c");
            System.Diagnostics.Process proc = System.Diagnostics.Process.GetCurrentProcess();
            Console.Out.Flush();
            proc.Kill();
        }

        static Int64 GetFileSize()
        {
            FileStream fs = null;
            Int64 length;
            try
            {
                fs = new FileStream(inArgs[0], FileMode.Open,
                        FileAccess.Read, FileShare.ReadWrite);
                length = fs.Length;
                return length;
            }
            finally
            {
                fs.Close();
            }
        }

        static void fileWatcher_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
        {
            DisplayUpdatedBytes();
        }

        public static void DisplayUpdatedBytes()
        {
            Int64 checkedSize = GetFileSize();
            if (checkedSize > currentFileSize)
            {
                string[] vals = { inArgs[0], (checkedSize - currentFileSize).ToString() };
                DisplayBytes(vals);
                currentFileSize = checkedSize;
                // Next line is test code
                //Console.WriteLine(string.Format("checkedSize : {0} currentFileSize {1}",checkedSize, currentFileSize));
            }
        }

        public static void DisplayBytes(string [] args)
        {
            if (currentFileType == filetype.None)
            {
                currentFileType = DetermineFileType();
            }

            FileStream fs = null;
            try
            {
                long numberOfBytesToDisplay = Convert.ToInt32(args[1]);

                // This opens the file for read, and does a FileShare that 
                // says the file can be opened for read / write by other processes.
                // This solves the problem of reading from a file that is already opened
                // by another process.  Very cool.
                fs = new FileStream(args[0], FileMode.Open,
                    FileAccess.Read, FileShare.ReadWrite);

                //ternary check insures that the attempted read of bytes is never more
                // than the total bytes in file.
                numberOfBytesToDisplay = numberOfBytesToDisplay < fs.Length ? numberOfBytesToDisplay : fs.Length;
                
                byte[] buf = new byte[numberOfBytesToDisplay];

                if (firstTime) // only display msg one time
                {
                    Console.WriteLine(string.Format("{0} = {1:#,#0} bytes.{2}", args[0], fs.Length, Environment.NewLine));
                    firstTime = false;
                }
                fs.Seek(-numberOfBytesToDisplay, SeekOrigin.End);
                fs.Read(buf, 0, (int)numberOfBytesToDisplay);

                // handles the conversion of the bytes (for output) according to the filetype
                switch (currentFileType)
                {
                    case filetype.Unicode:
                        {
                            buf = System.Text.Encoding.Convert(Encoding.Unicode, Encoding.ASCII, buf);
                            break;
                        }
                    case filetype.UTF8:
                        {
                            buf = System.Text.Encoding.Convert(Encoding.UTF8, Encoding.ASCII, buf);
                            break;
                        }
                    case filetype.Big:
                        {
                            buf = System.Text.Encoding.Convert(Encoding.BigEndianUnicode, Encoding.ASCII, buf);
                            break;
                        }
                }
                StringBuilder sbOutstring = new StringBuilder();
                foreach (byte b in buf)
                {
                    sbOutstring.Append(Convert.ToChar(b));
                }
                // changed line to .Write from WriteLine so I do not add the newline
                Console.Write(sbOutstring);
                currentFileSize = fs.Length;
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                ExitProc();
            }
            finally
            {
                if (fs != null)
                {
                    fs.Close();
                }
            }
        }
    }
} 

Points of Interest

Works across the network too. 

That means you can use UNC paths to tail files: \\ExtraServer\Thing1\code.dat

History 

First release of version 1.2.0.0 on 02/07/2013

Second release, version 1.3.0.0 on 04/29/2013

Third release version 1.3.1.0 on 4/30/2013 ### BUG FIX #### Since I am now reading the BOM characters out of the target file, I forgot to include the open for share indicator when opening the FileReader.  That means the app. fails when attempting to open a file that is already in use by another file.  Fail!  I fixed that in this version. I apologize for any inconvenience. 

Version 1.3.0.0 includes changes to support Unicode (UTF16), UTF8, Big Endian files in addition to the previously supported ASCII files. 

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

newton.saber
Architect
United States United States
My newest book is Learn Python, Think Python (amazon link opens in new window/tab)
 
My previous book is Object-Oriented JavaScript (See it at Amazon.com)
 
My book, Learn JavaScript - amazon.com link is available at Amazon.
 
My upcoming book, Learn AngularJS - Think AngularJS, will be releasing later in 2014.
 
You can learn more about me and my other books, at, NewtonSaber.com
Follow on   Twitter

Comments and Discussions

 
Questiongreat utility, thanks for sharing Pinmemberrazorrifh27-Mar-14 18:18 
AnswerRe: great utility, thanks for sharing Pinmembernewton.saber28-Mar-14 1:56 
SuggestionI vote 5 and request enhancement PinmemberDave Vroman3-Jan-14 5:49 
GeneralMy vote of 5 Pinmemberjoloda4-Jul-13 4:57 
GeneralRe: My vote of 5 Pinmembernewton.saber4-Jul-13 12:12 
QuestionThank you very much!!!!! PinmemberJustMe4TheCodeProject27-Jun-13 2:54 
AnswerRe: Thank you very much!!!!! Pinmembernewton.saber30-Jun-13 9:05 
Questionbaretail Pinmemberlqlau30-Apr-13 6:34 
GeneralMy vote of 5 PinmemberPrasad Khandekar29-Apr-13 19:29 
GeneralRe: My vote of 5 Pinmembernewton.saber10-May-13 7:45 
QuestionIssues with Unicode .. Code and fix here [modified] Pinmembermakerofthings711-Mar-13 8:43 
AnswerRe: Won't work with Unicode .. Code and fix here Pinmembernewton.saber13-Mar-13 2:26 
GeneralRe:Issues with Unicode .. Code and fix here Pinmembermakerofthings713-Mar-13 9:09 
GeneralRe:Issues with Unicode .. Code and fix here Pinmembernewton.saber29-Apr-13 15:26 
GeneralMy vote of 5 [modified] Pinmemberpt14019-Feb-13 22:28 
GeneralRe: My vote of 5 Pinmembernewton.saber10-Feb-13 7:11 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140926.1 | Last Updated 9 Dec 2013
Article Copyright 2013 by newton.saber
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid