Simple log parsing using MS Log Parser 2.2, in C#.NET

Introduction

At some stage in developing a server side application, I needed a way to check a particular user's download bandwidth. We have lots of client accounts in IIS6 and WMS (Windows Media Server), and each user has got a specific folder (like http://blablabla.com/xxx/, where xxx is a number like 122, 133 etc.). So if someone downloads media content from a specific user's directory, it will get logged in the IIS and WMS logs (IIS will log the download, and WMS logs the streaming bandwidth).

I wrote an NT service in C# to periodically parse the IIS/WMS logs and fetch the bandwidth usage of each user, and log it to a database. The code snippet is simple and easily understandable, and I suggest you download Microsoft Log Parser 2.2 freely from Microsoft, and check the samples and example code and SQL statements. It has got tons of features.

The sample log file I used is as below. I've put in bold, the directory name for which the bandwidth usage will be retrieved. (The text gets wrapped.. just manage guys).

#Version: 1.0
#Fields: date time cs-method cs-uri cs-version c-ip x-distributor-id 
x-ext-status sc-bytes time-taken sc-status cs(User-Agent) cs(Referer) cs(Cookie) 
cs-uri-stem cs-uri-query
2006-03-10 21:19:15.382 GET /Temp/59/Media/movies.wmv HTTP/1.1 
24.90.114.54 2011323 2000101 2018995 7393 206 "NSPlayer/9.0.0.3250 WMFSDK/9.0" 
"-" "-" /Temp/59/Media/movies.wmv -
2006-03-10 21:19:36.068 GET /Temp/59/Media/movies.wmv HTTP/1.1 
24.90.114.54 2011321 10 2020455 27 304 "Windows-Media-Player/9.00.00.3344" "-" 
"-" /Temp/59/Media/movies.wmv -
2006-03-10 21:19:36.084 GET /Temp/59/Media/movies.wmv HTTP/1.0 
206.24.192.232 2011323 121 2020455 1891 200 "Windows-Media-Player/9.00.00.3344" 
"-" "-" /Temp/59/Media/movies.wmv -

Using the code

Download and install Log parser 2.2 from Microsoft (Google it to find the download link.... very strange, it's been hidden somewhere deep in the MS site). After installing it, add a reference to LogParser.dll in the installation directory. Don't forget to put using MSUtil; on top.

//The passing argument "userID" is just the 

//folder name for the bandwidth used need to be retrieved.

//This is just for demonstration. 

//Re structure the SQL query for your needs.


public double ParseW3CLog( string userID )
{

    // prepare LogParser Recordset & Record objects

    ILogRecordset rsLP = null;
    ILogRecord rowLP = null;

    LogQueryClassClass LogParser = null;
    COMW3CInputContextClassClass W3Clog = null;

    double UsedBW = 0;
    int Unitsprocessed;
    double sizeInBytes;

    string strSQL = null;

    LogParser = new LogQueryClassClass();
    W3Clog = new COMW3CInputContextClassClass();

    try
    {
        //W3C Logparsing SQL. Replace this SQL query with whatever 

        //you want to retrieve. The example below 

        //will sum up all the bandwidth

        //Usage of a specific folder with name 

        //"userID". Download Log Parser 2.2 

        //from Microsoft and see sample queries.


        strSQL = @"SELECT SUM(sc-bytes) from C:\\logs" + 
                 @"\\*.log WHERE cs-uri-stem LIKE '%/" + 
                 userID + "/%' ";
    
        // run the query against W3C log

        rsLP = LogParser.Execute(strSQL, W3Clog);
    
        rowLP = rsLP.getRecord();
    
        Unitsprocessed = rsLP.inputUnitsProcessed;

        if (rowLP.getValue(0).ToString() == "0" || 
            rowLP.getValue(0).ToString() == "")
        {
           //Return 0 if an err occured

           UsedBW = 0;
           return UsedBW;
        }

        //Bytes to MB Conversion

        double Bytes = Convert.ToDouble(rowLP.getValue(0).ToString());
        UsedBW = Bytes / (1024 * 1024);

        //Round to 3 decimal places

        UsedBW = Math.Round(UsedBW, 3);
    }
    catch
    {
           throw;
    }

    return UsedBW;
}

Points of Interest

Quite easy, without lines and lines of code... and you will get all sorts of examples from the downloaded toolkit itself. This can be used to retrieve usage stats/bandwidth usage/file type/browser info.. you name it... from almost all types of standard log files (IIS/Mail/WMS/Apache...).