Click here to Skip to main content
15,350,147 members
Articles / Programming Languages / C#
Posted 19 Sep 2005


354 bookmarked

A Portable and Efficient Generic Parser for Flat Files

Rate me:
Please Sign up or sign in to vote.
4.93/5 (108 votes)
2 Aug 2017MIT8 min read
GenericParser is a C# implementation of a parser for delimited and fixed width format files.


We, as developers, are often faced with converting data from one format to another. For a project at work, I needed a portable solution that was efficient, had minimal external requirements, and parsed delimited and fixed-width data. As shown below, the GenericParser is a good replacement for any Microsoft provided solution and provides some unique functionality. The code is well organized and easy to follow to allow modification as necessary.

Note: The project was built using Visual Studio 2010, but the code is designed for .NET 2.0.


  • Delimited data - Data whose columns are separated by a specific character (e.g., CSV - Comma Separated Values).
  • Fixed-width data - Data whose columns are of a set number of characters wide.


The GenericParser (and the derived GenericParserAdapter) contains the following features:

  • Efficient - See Benchmarking below for more details.
    • Time: Approximately 3 to 10 times faster than any Microsoft provided solution
    • Memory: Approximately equal to or less than any Microsoft provided solution
  • Supports delimited and fixed-width formats
  • Supports a custom delimiter character (single character only)
  • Supports comment rows (single character marker)
  • Supports escape characters (single character only)
  • Supports a custom text qualifier to allow column/row delimiters to be ignored (e.g., multi-line data)
  • Supports escaped text qualifiers by doubling them up
  • Supports ignoring/including rows that contain no characters
  • Supports a header row
  • Supports the ability to dynamically add more columns to match the data
  • Supports the ability to enforce the number of columns to a specific number
  • Supports the ability to enforce the number of columns based on the first row
  • Supports trimming the strings of a column
  • Supports stripping off control characters
  • Supports reusing the same instance of the parser for different data sources
  • Supports TextReader and String (the file location) as data sources
  • Supports limiting the maximum number of rows to read
  • Supports customizing the size of the internal buffer
  • Supports skipping rows at the beginning of the data after the header row
  • Supports XML configuration which can be loaded/saved in numerous formats
  • Supports access to data via column name (when a header row is supplied)
  • Supports Unicode encoding
  • GenericParserAdapter supports skipping rows at the end of the data
  • GenericParserAdapter supports adding a line number to each row of output
  • GenericParserAdapter supports the following outputs - XML, DataTable, and DataSet
  • Thorough unit testing - 91.94% code coverage (tests supplied in source download)
  • Thorough XML documentation in code (including a .chm help file in the binary/source downloads)


To benchmark the GenericParser, I chose to compare it to:

  • Microsoft's Text Driver
  • Microsoft's Text Field Parser
  • Sebastien Lorion's CsvReader 3.7 (CSV only - code found here[^])
  • GenericParser 1.0

To get a realistic datasource for benchmarking, I took 10 rows of data from the Northwind Database and replicated them for successively larger and larger sets of CSV and FixedWidth data. Using System.Diagnostics.Stopwatch to measure CPU usage, I executed each benchmark 10 times and averaged the results to minimize the amount of error in the instrumentation. For the memory usage, I used Visual Studio 2010's memory profiling and executed each benchmark only once.

I've tried to generate tests that exercise the code equally for each solution. As a caveat, these tests do not test every possible scenario - your mileage may vary. Please feel free to use my code as a basis (or create your own tests) to compare the code before you draw any conclusions.

For example, the tests below did not take into account escaped characters. In GenericParser 1.0, it allocated an additional buffer for escaped characters, which essentially doubled its memory requirements. In GenericParser 1.1, it reuses the existing buffer to unescape the column of data. You wouldn't see this benefit unless you specifically geared your tests to account for this.

Just because I know someone will comment about this, I am aware of FileHelpers[^], but I believe they fit into a different category which doesn't map easily for comparison to the above solutions. FileHelpers rely on a declarative definition of the file schema through attributes on concrete classes.  My solution depends on defining the schema through properties or XML.   You may feel free to compare them, if they fit into your problem space.

CPU Usage

Image 1

Image 2

Memory Usage

Image 3

Image 4

Note: Because profiling the memory was generating .vsp files upwards of 2 gigs and the memory usage seemed pretty stable, I only executed memory profiling for 10 to 10,000 rows of data.


As can be seen in the charts, GenericParser meets or exceeds anything Microsoft has put together in all areas. Furthermore, version 1.1 out performs version 1.0 in performance considerably, especially considering the bug fixes and the new features added.

As can be seen by the graphs, Sebastien Lorion's CsvReader is definitely the top contender for parsing delimited files. So, if you are looking at only parsing delimited files, I would highly recommend checking out his library. Otherwise, I find my library to be an effective implementation for being able to parse both formats.

In the source download, you can find all of my performance tests and results, including an Excel 2010 workbook that has all of the collected raw data together for charting purposes.

Using the Code

The code itself mimics most readers found within the .NET Framework, but the usage follows four basic steps:

  1. Set the data source through either the constructor or the SetDataSource() method.
  2. Configure the parser for the data source's format, either through properties, or by loading an XML CONFIG file via the Load() method.
  3. Call the Read() method and access the columns of data underneath, or for the GenericParserAdapter - GetXml(), GetDataTable(), GetDataSet() to extract data.
  4. Call Close() or Dispose().
DataSet dsResult;
// Using an XML Config file. 
using (GenericParserAdapter parser = new GenericParserAdapter("MyData.txt"))
    dsResult = parser.GetDataSet();
// Or... programmatically setting up the parser for TSV. 
string strID, strName, strStatus;
using (GenericParser parser = new GenericParser())
    parser.ColumnDelimiter = "\t".ToCharArray();
    parser.FirstRowHasHeader = true;
    parser.SkipStartingDataRows = 10;
    parser.MaxBufferSize = 4096;
    parser.MaxRows = 500;
    parser.TextQualifier = '\"';
    while (parser.Read())
      strID = parser["ID"];
      strName = parser["Name"];
      strStatus = parser["Status"];
      // Your code here ...
// Or... programmatically setting up the parser for Fixed-width. 
  using (GenericParser parser = new GenericParser())
    parser.ColumnWidths = new int[4] {10, 10, 10, 10};
    parser.SkipStartingDataRows = 10;
    parser.MaxRows = 500;
    while (parser.Read())
      strID = parser["ID"];
      strName = parser["Name"];
      strStatus = parser["Status"];
      // Your code here ...


While I did not create a derivative of Sebastien Lorion's CsvReader, I did use some of his concepts of provided functionality in his CsvReader for the GenericParser.

Tools Used


  • September 17, 2005 - 1.0 - First release
  • June 20, 2010 - 1.1
    • New features:
      • Supports ignoring/including blank rows of data (no characters found in row)
      • Supports the ability to enforce the number of columns based on the first row
      • Supports stripping off control characters
      • GenericParserAdapter supports skipping rows at the end of the data
      • Reduced memory overhead when using escaped characters
      • Support for specifying the data's encoding
    • Bug fixes:
      • Fixed a bug with parsing data with a header and no data
      • Fixed a bug in not handling text qualifiers/escape/comment characters consistently
      • Fixed a bug in reading a file across a high latency network
      • Fixed a bug with text qualifiers being interpreted in the middle of the column (only works if at the start and end of a column)
      • Fixed a bug with skipping row ends at the very end of the buffer
    • Breaking changes:
      • Fixed width parsing will no longer take text qualifiers or escape characters into account
      • The following properties have been converted to a char?:
        • ColumnDelimiter
        • CommentCharacter
        • EscapeCharacter
        • TextQualifier
      • RowDelimiter has been removed, and the code automatically handles looking for '\n' or '\r' to indicate a new row (assuming '\r' is not a column delimiter). If one of these characters is found, it will skip the paired '\n' or '\r' (assuming '\r' is not a column delimiter).
      • SkipDataRows has been renamed to SkipStartingDataRows
      • The FixedWidth property has been replaced by a property called TextFieldType, which is of the enum type FieldType
      • Due to the changes in the properties listed above, the XML produced by version 1.0 will not be 100% compatible with version 1.1
      • Read() will return true if it parses a header row and no data
      • ParserSetupException has been replaced by InvalidOperationException
      • Reworked the messages supplied in the exceptions to be more descriptive.
  • June 26, 2010 - 1.1.1
    • New features:
      • Reworked benchmarking to be more representative of real world data and switched over testing to not use DataSets
      • Slightly more efficient loading of configuration files
  • February 5, 2012 - 1.1.2
    • Bug fixes:
      • Fixed an issue where an exception was being thrown for the MaxBufferSize was too small, when it was indeed large enough (Reported by uberblue).
  • March 16, 2012 - 1.1.3
    • Bug fixes:
      • Fixed an issue where control characters were being removed accidently (Reported by John Voelk).
      • Fixed an issue where data at the end of the stream wasn't extracted properly (introduced in version 1.1.2).
  • October 14, 2017 - 1.1.4
    • Bug fixes:
      • Fixed a minor issue with FileRowNumber when an exception was throw during parsing (Reported by webdevx).
  • October 14, 2017 - 1.1.5
    • Bug fixes:
      • Minor update to resolve issue with GenericParserAdapter that prevented adding columns if no data rows were found. (Reported by tschlarm).


This article, along with any associated source code and files, is licensed under The MIT License


About the Author

Andrew Rissing
United States United States
Since I've begun my profession as a software developer, I've learned one important fact - change is inevitable. Requirements change, code changes, and life changes.

So..If you're not moving forward, you're moving backwards.

Comments and Discussions

GeneralMy vote of 5 Pin
Hyland Computer Systems3-Mar-21 7:36
MemberHyland Computer Systems3-Mar-21 7:36 
QuestionUsing SkipStartingDataRows with FirstRowHasHeader? Pin
lauren meyers16-Aug-19 6:04
Memberlauren meyers16-Aug-19 6:04 
QuestionWhat to do if i have ' , '( comma) in the value? Pin
Member 1405666227-Mar-19 1:58
MemberMember 1405666227-Mar-19 1:58 
QuestionSAP idoc Pin
Damithaw13-Feb-19 21:35
MemberDamithaw13-Feb-19 21:35 
QuestionTrying to Read in a csv file into the a DataTable Pin
Member 1413135927-Jan-19 7:07
MemberMember 1413135927-Jan-19 7:07 
QuestionSupport for large file? Pin
FurqanSafdar29-Dec-18 20:14
professionalFurqanSafdar29-Dec-18 20:14 
QuestionError: Cannot implicitly convert type 'char[]' to 'char?' Pin
bekiser14-Nov-18 7:41
Memberbekiser14-Nov-18 7:41 
AnswerRe: Error: Cannot implicitly convert type 'char[]' to 'char?' Pin
Andrew Rissing15-Nov-18 3:06
MemberAndrew Rissing15-Nov-18 3:06 
GeneralRe: Error: Cannot implicitly convert type 'char[]' to 'char?' Pin
bekiser15-Nov-18 3:48
Memberbekiser15-Nov-18 3:48 
Questionconfiguration Pin
Member 1398207612-Sep-18 4:40
MemberMember 1398207612-Sep-18 4:40 
AnswerRe: configuration Pin
Andrew Rissing15-Nov-18 3:09
MemberAndrew Rissing15-Nov-18 3:09 
AnswerRe: configuration Pin
Andrew Rissing15-Nov-18 3:10
MemberAndrew Rissing15-Nov-18 3:10 
QuestionMaxRows for one time use? Pin
Member 1342051724-Aug-18 16:10
MemberMember 1342051724-Aug-18 16:10 
AnswerRe: MaxRows for one time use? Pin
Andrew Rissing15-Nov-18 3:12
MemberAndrew Rissing15-Nov-18 3:12 
BugBUG? FirstRowHasHeader skips first Data row! Pin
Freek Bos15-Aug-18 23:38
MemberFreek Bos15-Aug-18 23:38 
GeneralRe: BUG? FirstRowHasHeader skips first Data row! Pin
Freek Bos16-Aug-18 0:08
MemberFreek Bos16-Aug-18 0:08 
GeneralRe: BUG? FirstRowHasHeader skips first Data row! Pin
Andrew Rissing15-Nov-18 3:27
MemberAndrew Rissing15-Nov-18 3:27 
QuestionError CS8002 - Assembly does not have a strong name Pin
Kevin Yochum31-May-18 3:03
MemberKevin Yochum31-May-18 3:03 
AnswerRe: Error CS8002 - Assembly does not have a strong name Pin
Andrew Rissing13-Jul-18 4:14
MemberAndrew Rissing13-Jul-18 4:14 
GeneralRe: Error CS8002 - Assembly does not have a strong name Pin
Andrew Rissing17-Jul-18 4:05
MemberAndrew Rissing17-Jul-18 4:05 
BugColumn name returned as "Column1". Pin
razkar12-Mar-18 7:26
Memberrazkar12-Mar-18 7:26 
GeneralRe: Column name returned as "Column1". Pin
Andrew Rissing13-Jul-18 4:15
MemberAndrew Rissing13-Jul-18 4:15 
QuestionUsing the GenericParsing.chm file Pin
Paul G. Scannell23-Feb-18 3:04
MemberPaul G. Scannell23-Feb-18 3:04 
AnswerRe: Using the GenericParsing.chm file Pin
wawojr26-Mar-18 6:33
Memberwawojr26-Mar-18 6:33 
GeneralRe: Using the GenericParsing.chm file Pin
Paul G. Scannell6-Mar-18 6:42
MemberPaul G. Scannell6-Mar-18 6:42 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.