Click here to Skip to main content
13,049,263 members (68,236 online)
Click here to Skip to main content


5 bookmarked
Posted 22 Oct 2013

Collect and Compare Log Statistics using LogJoin

, 22 Oct 2013
The LogJoin tool helps to collect any unstructured data from text files and join it to a simple table representation for easy analysis.
<?xml version="1.0" encoding="utf-8" ?>
   Copyright (c) 2013, Yuriy Nelipovich

   If you find this application useful or in case of any questions, suggestions
   bug reports, donation, please email me:

    <section name="parameters" type="LogJoin.Config.Parameters, LogJoin"/>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />

  <!-- Configuration of the application is below -->
    <!-- <expressions> element contains a list of regular expressions used to parse content of input files. -->
      <!-- An expression is defined in <expr> element. You should specify an unique name for each expression.
           Expression can be multiline, this means that it can match multiple lines at once (see System.Text.RegularExpressions.RegexOptions.Multiline for details).
           When parsing input text, expression extracts a set of values and then the values are stored to the Record. Each new match of the expression produces new Record.
           To capture the values the expression should define named groups. In the example below, the group (?<messageAuthor>\w+) will capture authors name.
           Some of the values should represent a unique key for the Record, see example below. -->
      <expr name="recipientRecord" multiline="true"><![CDATA[^New message received, ID: (?<messageID>\d+), sender: (?<messageAuthor>\w+)\. Processing it.\r\nTitle: (?<title>[\w ]+)\r\nCurrent time: (?<timeReceived>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\r\nIs spam: (?<isSpam>\w+)\r\n]]></expr>
      <expr name="senderRecord" multiline="false"><![CDATA[^Meesage sent from account (?<messageAuthor>\w+)\. Message ID: (?<messageID>\d+), time: (?<timeSent>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}); title: (?<title>[\w ]+); words count: (?<wordsCount>\d+)]]></expr>
    <!-- <inputs> element contains a list of inputs that provide data for processing -->
      <!-- Input is a text file (or files) containing values for Records. The input should have unique name 
           taht has similar meaning as table name in sql queries.
           "lines" attribute defines how many lines of text in input file may correspond to one Record. In other words
           this is maximum count of lines that the regular expression can match. -->
      <input name="recipient" lines="5">
        <!-- Path to the directory containing the input file (or files) -->
        <!-- Name of the input file (or file mask for multiple files) -->
        <!-- Name of the regular expression (defined above) that will be used to parse the input -->
        <!-- Set of column names (fields) that each Record contains. These names must correspond to named groups defined for the expression -->
        <columns>timeReceived, messageAuthor, messageID, isSpam, title</columns>
        <!-- Set of values that together form a unique key for each Record. Order of key values is important.
             This key is used in Left Join operation between different inputs, see details below. -->
        <key>messageAuthor, messageID</key>
      <!-- The second input produces set of records for left side of Join operation -->
      <input name="sender" lines="1">
        <!-- This input contains a set of text files, they are ordered by file name -->
        <columns>messageAuthor, messageID, timeSent, title, wordsCount</columns>
        <!-- Keys of each input should have the same order and the same format
             because when joining Records from different inputs the key values are compared as strings. -->
        <key>messageAuthor, messageID</key>
    <!-- Path to output file. The file contains a table in csv format.
         File name can have string format argument that has DateTime value (default is current date and time) -->
    <!-- Delimiter for values in output file -->


By viewing downloads associated with this article you agree to the Terms of Service and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.


This article, along with any associated source code and files, is licensed under The BSD License


About the Author

Yuriy Nelipovich
Software Developer CactusSoft
Belarus Belarus
No Biography provided

You may also be interested in...

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.170713.1 | Last Updated 22 Oct 2013
Article Copyright 2013 by Yuriy Nelipovich
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid