Click here to Skip to main content
Click here to Skip to main content

ASC2XXX - Two classes for parsing delimited text files

, 8 Mar 2003
Rate this:
Please Sign up or sign in to vote.
Convert delimited text files to XML file or DataSet object
<!-- Add the rest of your HTML here -->

Turn a delimited text file:

Into a DataSet:

Or an XML file:

Introduction

Two classes that illustrate one way to: read a delimited text file, parse the "fields" of data using regular expressions and move the data it into either an XML file or a DataSet object for direct use. 

.Net framework classes used:

  • System;//For strings and things
  • System.IO;//For reading and writing streams and files
  • System.Xml;//For creating and writing the XML file
  • System.Text.RegularExpressions;//For parsing the text file
  • System.Data;//to generate a DataSet

Concepts illustrated

  • Reading and writing files through stream objects
  • Parsing text using regular expressions
  • Generating a DataSet in memory from code and using it to fill a DataGrid control
  • Generating an XML file from code

Background

The reason I wrote these classes is twofold:

  • I needed to write an application that would parse a web server log file (in W3C common log format) and put that data into a SQL server database. 
  • I needed a class that I could re-use in other applications where it was necessary to move data from a CSV text file into a database.

Using the code

Although very short, the code is commented heavily throughout and contains referenced hyperlinks to the MSDN articles that explain in more detail the .Net class being used at each point in the code where relevant.

This code is set to parse a web server log file, however it can easily be modified to parse any delimited text file and I've indicated in the comments where to do so. I've also included a commented out line of an alternate regular expression that can be used to parse comma delimited text files.

A file samplelog.txt is provided with the demo which contains a test web server log file.  I have mangled the IP Addresses for privacy, however the data is straight out of an Apache server log from our web server.

I've recently started using C# after many years of working in C++ so any constructive criticism would be welcome.

History

  • Original version: Feb.26.2003

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Member 96

Canada Canada
No Biography provided

Comments and Discussions

 
GeneralGood Work PinmemberDharmarajNagarajan16-Feb-09 23:36 
GeneralMy vote of 1 PinmemberHoward Richards29-Jan-09 22:10 
Questionhow to convert the logfile into database Pinmemberbhavna81628-May-07 21:29 
GeneralI hit a bug PinmemberKevinRMacDonald9-Feb-07 8:02 
GeneralRe: I hit a bug PinmemberJohn Cardinal9-Feb-07 8:17 
AnswerRe: I hit a bug PinmemberKevinRMacDonald9-Feb-07 11:14 
GeneralRe: I hit a bug PinmemberJohn Cardinal9-Feb-07 11:48 
QuestionWhy not use an Array? Pinmemberkheiligh6-Feb-07 4:05 
AnswerRe: Why not use an Array? PinmemberJohn Cardinal9-Feb-07 12:00 
GeneralGreat Work! PinmemberPengie28-Nov-06 14:44 
GeneralExcellent works right away (Here are some changes I did to use DataGridView and remove Carriage Return and Line Feeds) PinmemberLuluSailor16-Aug-06 6:39 
QuestionNewbie - any updates regarding use of ADO.NET 2.0? PinmemberLuluSailor16-Aug-06 4:15 
AnswerRe: Newbie - any updates regarding use of ADO.NET 2.0? PinmemberJohn Cardinal16-Aug-06 6:07 
GeneralRe: Newbie - any updates regarding use of ADO.NET 2.0? PinmemberLuluSailor16-Aug-06 6:29 
GeneralNicely done PinmemberSharpmike19-Apr-06 3:43 
Generalparsing out certain data in text file Pinmemberjohnnyphaze1-Dec-05 10:14 
GeneralRe: parsing out certain data in text file PinmemberJohn Cardinal2-Dec-05 3:02 
Hello, the code illustrates two ways of parsing a text file, one is to an xml document the other to a dataset, either one could be saved to file.
 
If you mean extracting certain fields and then rewriting out another delimited text file with just those fields, I would probably use the dataset example as a starting point to write a program that allowed a user to scan the first row of the input data, use those fields or field names as a basis to build a user interface selection row for the user so they can select the fields they want to import, probably I would just import all the data without skipping any fields into a dataset as in the example, then when writing it back out to a text file would be the point I would skip the unrequired fields. That way you can check that the data source is "clean" i.e. self consistent, no missing fields etc without trying to pick and choose out the fields in the import part of the process. (unless of course we're talking about gigabyte sized files or something)
 
Although if you are in the habit of working with data in text files then you may want to consider the XML version since that's a natural fit for textual data storage.
 
If you are looking at writing an application that will repeatedly deal with the same format text file and repeatedly write out another format text file as a result then you could probably use much of the code in the example with very little changes other than redefining the fields as you require them from my old apache web log fields to whatever works for you see comments in source code about doing exactly that.
 
If you just need to do ad-hoc data transformation of text files I wouldn't bother writing a program for that, there are so many out there, Microsoft Access is excellent for general purpose data conversion.

GeneralCSV Files Pinmemberboomer194616-Aug-05 18:28 
AnswerRe: CSV Files PinmemberJohn Cardinal17-Aug-05 5:45 
GeneralRe: CSV Files Pinmemberboomer194617-Aug-05 16:21 
Generalshows ur Xperience PinmemberSrinivas Varukala20-Jun-05 7:55 
GeneralThank you. Pinmemberm9u3523-Jan-05 9:44 
QuestionWhy not use a normal data connection? PinmemberPaul Menefee10-Mar-04 4:19 
AnswerRe: Why not use a normal data connection? PinmemberJohn Cardinal10-Mar-04 5:14 
GeneralRe: Why not use a normal data connection? PinmemberPaul Menefee10-Mar-04 5:51 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140821.2 | Last Updated 9 Mar 2003
Article Copyright 2003 by Member 96
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid