Click here to Skip to main content
15,867,704 members
Articles / Security

Poor Man Web Monitoring Tools

Rate me:
Please Sign up or sign in to vote.
4.00/5 (1 vote)
13 Aug 2018CPOL10 min read 8K   15  
This article shares the idea on how to utilize free tools to download log files from server, upload IIS log files into the database, display the results and compare the baseline and production files.

Table of Contents

Background

This project was initially started back in 2012. Then, one day, my computer hard drive crashed. I was lucky enough to be able to recover most of the files. After spending some time, I was able to get it to compile again. After that, I work on it maybe once or twice a year. There is no doubt, lots of improvement can be made to it. Lately, I've made some changes to the project and decided to share it now, instead of letting it sit on the shelf for another couple of years. Hopefully, someone will find this project useful and continue to make enhancements to it.

Introduction

Like many other bloggers or website owners out there, most of us host our website on a shared server instead of dedicated server hosting due to the cost. The hosting company shall not be named here. Based on my own experiences, once in a while, some malicious folders and files / malware were being injected into the website. Every time I submitted a ticket about it, I got a response like "your computer was compromised", "you're using outdated software", "you need to change your password", etc. All kind of nonsense, and the hosting company never took responsibility or do their due diligence. Some hosting companies even offer to clean up the vulnerability and monitor EACH site on the host for a yearly subscription. Imagine if you have more than one website, the cost will quickly add up due to someone else's negligence.

Since I refused to pay some company every year to monitor all my websites, why not build my own? The title said it all, "Poor Man Web Monitoring tools", if you're poor, then you need to work hard and put together all the available free tools yourself. It might look like a Frankenstein tool now, but I’m positive it will look better once we pour some more thought into it.

Shown below is the brief description of some of the components in the solution. In this article, I'll not go into every detail on how the solution was being implemented as I believed some more work is needed to optimized it. But I will share what each component will do and how to set it up.

  1. DownloadIISLog
    • Download IIS Log files thru FTP (WinSCP)
    • Insert the log files data into the SQL Database using Log Parser
  2. SimpleMonitor.Web
    • Web UI to display the IIS Log data using jqGrid
    • Option to mark certain Ip Addresses "Banned"
  3. SimpleMonitor.BlockIpAddress
    • Example of HTTP Module to block banned Ip address from accessing the Website
  4. SimpleMonitor.ScanFile
    • Download latest files
    • Compare Baseline vs latest files using WinMerge
    • Send email notification of the results

DownloadIISLog

One of the responsibilities of this console application is to download the logs file from the FTP server using WinSCP. Please note that, I only tested this on two different shared hosting environments. Here is what I would suggest if you get permission denied error while accessing the log folder. First, login to your hosting account, navigate to the file manager, look for the logs folder and assign read only permission to the folder. The second option is to submit a ticket.

Shown in listing 1 is the method to download the log files. The initial run will take a while depending on the amount of log files residing in the hosting log folder. The subsequent execution will be a lot faster because this method will download only files not previously downloaded before. To achieve this, the method will first get the list of log files metadata from the server using the WinSCP ListDirectory method and store it in the directoryInfo object. Next, it will get the most recent log file name stored in the database. After that, the module will try to query the directoryInfo object for the LastWriteTime by the log file name. Finally, the module will download all the log files where the LastWriteTime is after the LastWriteTime that was determined previously.

Listing 1

C#
static internal void WinScpGetLog() {
 using(Session session = new Session()) {
  // Connect
  session.Open(GetWinScpSession());

  string remotePath = ApplicationSetting.FtpRemoteFolder;
  string localPath = ApplicationSetting.DownloadPath;

  // Get list of files in the directory
  RemoteDirectoryInfo directoryInfo = session.ListDirectory(remotePath);

  //latest file name in the table by application name
  var latestLogFileInDb = Path.GetFileName
       (unitOfWork.IISLogRepository.LatestFileName(ApplicationSetting.ApplicationName));

  //get the date of the latest file from FTP
  var logFileDate = directoryInfo.Files
   .Where(w => w.Name.ToLower() == latestLogFileInDb ? .ToLower())
   .Select(s => s.LastWriteTime).FirstOrDefault();

  // Select the files not in database table
  IEnumerable notInLogTable =
   directoryInfo.Files
   .Where(file => !file.IsDirectory && file.LastWriteTime > logFileDate).ToList();

  //// Download the selected file
  foreach(RemoteFileInfo fileInfo in notInLogTable) {
   string localFilePath =
    RemotePath.TranslateRemotePathToLocal(
     fileInfo.FullName, remotePath, localPath);

   string remoteFilePath = RemotePath.EscapeFileMask(fileInfo.FullName);

   //download
   TransferOperationResult transferResult =
    session.GetFiles(remoteFilePath, localFilePath);
  }
 }
}

Listing 2 shows the logic employ the Microsoft Log Parser tool to query the log files and insert it into the database. The query use by the Log Parser in CallLogParser() method is very straight forward. It will connect to the SQL database using the provided username and password. Then read all the log files in the folder and then insert them into a table. The ApplicationName parameter was added recently as an option to allow storing log files data for more than one website. If you have ten websites, you can duplicate this console application ten times and configure different ApplicationName value in the configuration file.

Listing 2

C#
 static void CallLogParser() {
 ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.LogParserPath);
 startInfo.WindowStyle = ProcessWindowStyle.Minimized;

 Process.Start(startInfo);

 startInfo.Arguments = string.Format("\"SELECT '{7}', *,1 INTO {0} FROM {1}{2}*.log\" 
      -o:SQL -createTable:OFF -server:{3} -database:{4} -username:{5} -password:{6}",
  ApplicationSetting.LogParserSqlTable, ApplicationSetting.DownloadPath, 
      ApplicationSetting.LogFilePrefix, ApplicationSetting.LogParserSqlserver,
  ApplicationSetting.LogParserSqlDatabase, ApplicationSetting.LogParserSqlUserName, 
      ApplicationSetting.LogParserSqlPassword,
  ApplicationSetting.ApplicationName);

 Process.Start(startInfo);
}

Shown in figure 1 is the data imported into the database table through the CallLogParser() method.

Figure 1

logparser insert into database

Figure 1

DownloadIISLog - Configuration

Table 1 shows the descriptions of the settings in the app.config.

Table 1

Key Description
FtpHost The address of the server. Example: poormantool.arr or 123.123.123.123
FtpRemoteFolder The path to the logs files on the server. Example: /virtualFolder/logs/W3SVC999/
FtpUserName FTP account
FtpPassword FTP password
FilePrefix The log file prefix, if any.
DownloadPath Location to store the downloaded logs file. Example: c:\temp\app1\download\
LogPath LogPath is the path to the log file for logging purposes like when the console run, stop, error, etc.
Example: c:\temp\app1\log\log.txt
LogParserPath The path to the log parser. Example: C:\Program Files (x86)\Log Parser 2.2\LogParser.exe
LogParserSqlTable The table to use in the SQL database to store the log files data. Right now, it can only be "iislog" unless you update the entity and code
ApplicationName The name of the application. Example: app1
LogParserSqlserver The SQL server address
LogParserSqlDatabase The SQL server database
LogParserSqlUserName The SQL server username
LogParserSqlPassword The SQL server password
connectionStrings Replace the XXX with your SQL server information

SimpleMonitor.Web

The purpose of this web application is to display the log files data. Now that we have the data in the database, it is up to you what technology you want to use to interface with the data. This web application was developed by using MVC 5, EntityFramework v6.0, jqGrid, Bootstrap v4.0 and jQuery v3.3.

Figure 2 shows how the Interface looks like from the web browser. All the columns are sortable and filterable. The blocked? Column indicates if the IP address is blocked from accessing the site. The goal of the blocked Hit column is to show number of times a blocked IP address attempts to access the website.

Display IISlog data in jqGrid

Figure 2

To block an IP address, click on the green icon, a confirmation dialog will appear, click Continue. Refer to figure 3. The application will insert the selected IP address into the dbo.BlockedIp table, then mark all the blocked IP in the grid with a red ban circle icon.

blocking ip addresses

Figure 3

To unblock an IP address, click on the red circle icon and continue button. Refer to figure 4.

unblocking ip addresses

Figure 4

SimpleMonitor.Web - Configuration

Make sure to update the connection string in the web.config to mirror your SQL server environment.

SimpleMonitor.BlockIpAddress

The purpose of this module is to demonstrate how to utilize the data in dbo.BlockedIp table. This module will check if the requester IP address exists in the table, if yes, redirect the request to an error page and then increase the blocked hit count. Listing 3 shows how to register the httpModules in the web application web.config file. To test it, I'll add my IP address into the table and re-run the web application. Again, this is just an example, it is up to you how you want to implement the detection and prevention control.

Listing 3

XML
<system.web>
   <httpModules>
      <remove name="BlockIpHttpModules" />
     <add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, 
                SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" />
    </httpModules>
  </system.web>
  <system.webServer>
    <modules>
      <add type="SimpleMonitor.BlockIpAddress.BlockIpHttpModule, 
       SimpleMonitor.BlockIpAddress" name="BlockIpHttpModules" preCondition="managedHandler" />
    </modules>
  </system.webServer>

SimpleMonitor.ScanFile

The main purpose of this console application is to compare the folder and files between the baseline and the production. Initially, the application will download and save the files to the destined folder using WinSCP. Then it will run the WinMerge command to compare the files with the baseline. Listing 4 shows the WinMerge command line to compare the files.

Listing 4

C#
static void CompareFiles() {
 var tempFileName = $ "{Guid.NewGuid()}.html";

 ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.WinMergePath);
 startInfo.WindowStyle = ProcessWindowStyle.Minimized;

 startInfo.Arguments = $ " {ApplicationSetting.BaselineFilesPath} 
                           {ApplicationSetting.LatestFilesPath} -minimize " +
  "-noninteractive -noprefs -cfg Settings/DirViewExpandSubdirs=1 -cfg Settings/DiffContextV2=2 " +
  "-cfg ReportFiles/ReportType=2 -cfg ReportFiles/IncludeFileCmpReport=1 -
                                  cfg Settings/ShowIdentical=0 " +
  $ " -r -u -or {ApplicationSetting.FileCompareOutputPath}{tempFileName}";

 var process = Process.Start(startInfo);
 process.WaitForExit();

 Email.SendEmail($ "{ ApplicationSetting.FileCompareOutputPath}{ tempFileName}");
}

Table 2 shows the WinMerge parameters in listing 4 and its descriptions.

Table 2

Key Description
-minimize starts WinMerge as a minimized window.
-noninteractive Exit WinMerge after compare / report generation
-noprefs Do not read / write setting information from registry (use default value)
-cfg Settings/DirViewExpandSubdirs=1 Expand folder tree after comparison 0: do not expand folder tree
-cfg Settings/DiffContextV2=2 0: shows only the different
1: shows the different and one line from above and below
2: shows the different and two lines from above and below
-cfg ReportFiles/ReportType=2 Generate HTML-format report
-cfg ReportFiles/IncludeFileCmpReport=1 Include file comparison report in folder comparison report, 0: do not include
-cfg Settings/ShowIdentical=0 Do not show Identical files in the result
-r compares all files in all subfolders
-u prevents WinMerge from adding either path (left or right) to the Most Recently Used (MRU) list
-or Path to output file

At the end of the comparison, the CompareFiles() method will send a summary to the specified email address using Gmail. Figure 5 shows how the comparison summary looks like.

Winmerge result summary

Figure 5

Figure 6 shows how the comparison details look like when clicking on the Filename to drill down from the computer where the comparison reports are saved.

Winmerge result details

Figure 6

SimpleMonitor.ScanFile – Configuration

Table 3 shows the descriptions of the settings in the app.config.

Table 3

Key Description
FtpHost The address of the server. Example: poormantool.arr or 123.123.123.123
FtpRemoteFolder The path to the logs files on the server. Example: /virtualFolder/wwwroot/
FtpUserName FTP account
FtpPassword FTP password
BaselineFilesPath The path to the application production files
LatestFilesPath The path to the latest files downloaded by using the WinSCP from the FTP Server
FileCompareOutputPath The path where the comparison results will be stored
WinMergePath The path to the WinMerge application
SmtpHost The email SMTP host
SmtpTo The recipient
SmtpPort The SMTP Port number
SmtpUserName The email account username / email
SmtpPassword The email account password
SmtpSubject The email subject

Required Software

Download and install the following software.

How to Setup

Execute the database_objects.sql script on an existing / new database instance. The script will create the following objects:

  • [BlockedIp] table
  • [iislog] table
  • [vwIISLog] view
  • [InsertUpdateBlockedIp] Stored Procedure

Conclusion

First of all, I would like to thanks the new WinMerge owner for continuing to maintain the WinMerge software. Please keep in mind that the solution provided here is not a preventive control. Whatever you see in the logs and comparison results are after the facts. But you can create your own preventive control such as IP blocker after reviewing and analyzing the information.

Here are other topics of interest that you can extract out of this project.

  • How to Use jqGrid with server-side paging, filtering, sorting
  • How to use Log Parser to import IISLog into SQL database table
  • How to use WinSCP to download files from FTP server
  • How to use WinMerge to compare files
  • How to create and utilize HTTP Modules
  • How to send email using Gmail

I hope someone will find this project useful. If you find any bugs or disagree with the contents or want to help improve this article, please drop me a line and I'll work with you to correct it. I would suggest visiting the demo site and explore it in order to grasp the full concept because I might miss some important information in this article. Please contact me if you want to help improve this article.

History

  • 08/12/2018 - Initial version

Download

Resources

Feedback/Issue

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Software Developer (Senior)
United States United States
I have over 10 years of experience working with Microsoft technologies. I have earned my Microsoft Certified Technology Specialist (MCTS) certification. I'm a highly motivated self-starter with an aptitude for learning new skills quickly.

Comments and Discussions

 
-- There are no messages in this forum --