Click here to Skip to main content
Click here to Skip to main content

FileHelpers v2.0 - Delimited (CSV) or Fixed Data Import/Export Framework

, 10 Jul 2007
Rate this:
Please Sign up or sign in to vote.
An easy to use .NET library to read/write strong typed data from files with fixed length or delimited records (CSV). Also has support to import/export data from different data storages (Excel, Acces, SqlServer, MySql)

FileHelpers Home Page (Project Summary)

FileHelpers v 2.0

Browse the Forums Development Blog Support the project

Introduction

The FileHelpers are an easy to use library to import/export data from fixed length or delimited flat files (like the CSV). FileHelpers also has support to import/export data from different data storages (Excel, Acces, SqlServer).

The library has a set of converters for the basic types and can be easily extended to provide custom converters.

The main idea is pretty simple:

You can strong type your flat file (fixed or delimited) simply describing a class that maps to each record of the file and work with your file like a strong typed .NET array.

Basic Uses

Directly between the files and the .NET source code:

Use 1

It also can be used as an intermediate between files and MS Access or MS SqlServer tables:

Use 2

Quick Start Guide (easy steps)

To start using the FileHelpers Library you only need to add a reference in your project to the file: FileHelpers.dll. You can find it in the Release directory of the distribution. Tip: remember to leave the FileHelpers.xml file to get Intellisense support.

Next you need to define a class that maps to the record in the source/detination file. For this example we use a file with this format delimited by a |:

10248|VINET|04071996|32.38
10249|TOMSP|05071996|11.61
10250|HANAR|08071996|65.83
10251|VICTE|08071996|41.34
...............

The class that we refer can be like this:

[DelimitedRecord("|")]
public class Orders
{
   public int OrderID;

   public string CustomerID;

   [FieldConverter(ConverterKind.Date, "ddMMyyyy")]
   public DateTime OrderDate;

   public decimal Freight;
}

Later you must instantiate a FileHelperEngine and Read/Write files:

FileHelperEngine engine = new FileHelperEngine(typeof(Orders));

/// to Read use:
Orders[] res = (Orders[]) engine.ReadFile("input.txt");

/// to Write use:
engine.WriteFile("output.txt", res);

Finally you can use the res array to access each item in the file, for example:

 foreach (Orders order in res)
 {
    Console.WriteLine("Order Info:");
    Console.WriteLine(order.CustomerID + " - " +
                      order.OrderDate.ToString("dd/MM/yy"));
 }

Working with FixedLength files is exactly the same but you need to use the [FixedLengthRecord] and [FieldFixedLength] attributes.

Record Class Wizard

Since version 1.3.5 the library has had a Record Class Wizard to generate the record class. The Record Class Wizard allows you to generate your record mapping class, save the definition, generate code based on snippets (Read File, Read with Generics, ReadAsync and so on).

The wizard uses the excellent FireBall CodeEditor


Click to see how it works

Who needs the File Helpers Library?

In almost every project there is a need to read/write data from/to a flat file of a specified format. Writing the code to process these files is not hard, you can use String.Split or String.SubString, but if you do that the code sometimes becomes hard to read and change. If you want to add support later for quoted string, multiline, convert types, etc. these things tend to get complicated.

So with the FileHelpers you don't need to worry about changes in the file structure because you can change the Type, Converters, Triming, Length in seconds.

Another place where you can find the FileHelpers useful is for log parsing, data warehouse and OLAP applications, communication between systems, file format transformations (for example from a fixed length to an CSV file), load test data into your NUnit tests and a lot more!!!

This library aims to provide an easy and reliable way to accomplish all these tasks.

Main Features

Easy to use: The FileHelpers Lib is straight forward to learn and use. (see EasyExample)

Auto Converters: The library has a set of converters for the basic types and can be easy extended to provide custom converters. (see ConverterBase and Converter Example)

RunTime Classes since version 1.6.0 you have been able to create your record class at run time, and load them from files with source code or an XML description (check the example)

Master-Detail: You can read and write records with a master/detail pattern. (see Example1 and Example2)

Multiple record format support: With the MultirecordEngine you can read files with different record layout, you can also read files with some delimited and some fixed length records. (see Example)

Event Support: The engines of the library contain some events to allow you to easily extend the behavior of the library (see Example)

MS Excel Storage: Are a way to extract / insert records between any source and an excel file. (see ExcelDataStorage and ExcelDataLinkExample)

DataLinks: Are a way to extract / insert records between a database and a file. (see DataStorage and FileDataLinkExample)

GenericDataLink: Now you can to copy records between two Data Storages (see GenericDataLink and DataStorage)

Asynchronous Mode: You can use the library to read line by line and not the whole file. (see Async Methods)

.NET Compact Framework Support: From the version 1.1 you can use the FileHelpers library for you PocketPC and WindowsCE developments. (thanks Pierre)

File Transform Engine: To convert files in one format to another (for example a file with CSV to a FixedLength record format) (check the example)

Progress Notification: To get notified of the progress in each operation in the library (check the example)

.NET 2.0 Generics: the cast less and strong typed version of the engines (check the example)

FileDiffEngine to allow compare files with the same record layout (check the example)

Others Features

Files, Stream and String Support: You can use the library to read/write any stream or string, not only files. (see FileHelperEngine)

Different Error Behaviors: You can set the behavior of the library when an error is found (throw and exception, ignore, save and continue, etc) (see the examples)

Quoted String Support: It allows to indicates that the field must be read and written as Quoted String, like Excel CSV. (see FieldQuotedAttribute )

CommonEngine a easy way to access to common operations (check the example)

Encoding Aware: You can define the encoding used to read and write files or streams. (see BaseEngine.Encoding)

NullValues: The library identifies the null values in the files and assigns an adequate value. (see FieldNullValueAttribute)

Good Documentation: The library is fully documented (at least that's the intention) with a lot of Examples of Use.

Align and Trimming: You can set the processing mode of the in/out strings with a lot of attributes like the FieldAlignAttribute and the FieldTrimAttribute.

Progress Notification

Here is a screenshot of the progress notification of the demo app.

Since versión 1.4.0 you can get notified of the progress of each operation of the library to show feedback to the user or whatever you want.

Class Diagram

Detailed Diagram

Detailed Diagram

Testing

The library contains more than 250 NUnit tests to ensure correctness:

Sample image

History

Version 2.0 (April-2007)

The library has passed his 1st birthday!!! Thank you all for your work and contributions (partial list of them)

A lot of things were changed (mostly internally), now we have a performance gain of more than 60% for .NET 2.0

I keep releasing the .NET 1.1 version because I know that a lot of people use it in some corportaions, but it could be that new features in the versions to come will be .NET 2.0 only. It is monumental work to maintain and optimize both versions.

There are also a lot of major changes, a lot of refactoring in the code, and a big core rewrite that allows the engines to use less memory and temporary strings (we are using more buffers to aviod this)

I keep working hard to keep the library updated and give support to the users. You can check for examples in the Forums which shows a lot of problems that have been solved. I also ask for a lot of mail with problems from the users.

We have our own domain www.filehelpers.com thanks to (Antoine) and a Development Blog

Breaking Changes

  • The constructor of the MultiRecordEngine was changed to allow params args, and to disallow users from passing the RecordSelector (in write operations were not needed).
  • EndsRead and EndsWrite deleted, now you have a Close() operation to simplify the API and to make it similar to the System.Data namespace. (The async engines also implement IDisposable)
  • Renamed FileHelperException --> FileHelpersException (a better name and an easy solution to a problem)
  • By default the numeric fields in the FixedLengthRecords that don't have a [FieldAlign] will be aligned to the right and the rest the left by default, with this we avoid the problem of generating files with different meaning when we read and write.

Performance Related

  • Internal use of Reflection.Emit, Dynamic Methods and char buffers to get more than a 50% enhace in performance and for .NET 2.0 more than 65%
  • The Read..AsDT methods now create the DataTable record by record, and not at the end so you can handle large files without any memory overload
  • Not more reflection in the operations (only in the constructors).
  • FieldSorter faster too, Removed REFLECTION for EMIT

Shining new features

  • The very cool DelimitedFileEngine and FixedFileEngine that allow to change options at RunTime (also with generic versions) Check an example here
  • ConditionalRecords you can easily include or exclude certain records based on a RecordCondition (like BeginsWith, EndsWith, Contains or RegEx) Check the example
  • A new GenericDatabaseStorage which is excellent for allowing others Db with ASO.NET support to work with the FileHelpers (Thanks Rodolfo Finochietti)
  • Experimental Mono Project Support (some users are already using it, we hope that in the next release we have a completely working version for this framework)
  • New Version Checker in the demos and Wizard (both with renew look and feel) Check the movies
  • .NET 2.0 Nullable Types support. (Thanks Vijayan) Check the example
  • Notification Interfaces: you can now get notified of the events simply implement INotifyRead or INotifyWrite
  • Enhaced Debugging in .NET 2.0 with DebuggerDisplay and DebuggerBrowsable, DebugVisualizars to come.
  • The Wizard has a Record Class Test that allows you to introduce sample data and your class and check for errors or results.
  • You can check some screencast about the demos, wizard and other features of the library.
  • Now the code for VS2003 and VS2005 are both on SVN, sharing the same files and the build script autodetect your Visual Studio installation. So no more excuses: time to get involved.
  • The engines with async ops are now IDisposable and IEnumerable so you can declare it with using(..) and use them in a foreach loop

API Changes and Extensions

  • The write methods are less strict. You can pass now an IEnumerable instead of an array, so you can direct pass a List or ArrayLIst to the method without needing to do a ToArray()
  • RunTime records can now get a DataTable in the constructor and so use DataColumns to create one field for each column.
  • Generic versions of near all the engines of the library (it's really hard to mantain the two copies so everyone, it's time to port!!)
  • Async Operations in others engines like the MultiRecordEngine
  • Event support to the MultiRecordEngine thanks to the contribution of Francis de Fouchier
  • The library handle now has infinite levels of inheritance adding the high level fields first in the result record class.
  • Allow to ignore spaces and tabs in the IgnoreEmptyLinesAttribute
  • IgnoreCommentedLines thanks to MCampbell
  • The Read methods now have a maxRecords args to tell to the engines how much they must read.
  • A new constructor overload to the engines that allows you to pass the Encoding, which is useful for making the users aware of the Encoding feature.
  • FileTransformEngine with two new methods ReadAndTransformRecords and TransformRecords
  • Flush Method to AsyncEngines to allow users to ensure data write, for example, if they are using the lib for logging.
  • The converters are now smarter. They validate to which types can be assigned and throw exceptions if it is wrong (for example if you use ConverterKind.Decimal in an int field)
  • The ConvertException has a lot of context information: LineNumber, ColumnNumber, FieldName. Also has a better Exception Message.
  • The integer converters now recieve a decimal separator to build an InvariantCulture to format and parse the values and strings

Small Changes

  • Fixed the problem with ASP.NET: "Line 0: Metadata file 'filehelpers.dll' could not be found"
  • Fixed some problems with the CsvClassBuilder (it's a good practice to not release features added last-minute)
  • The Assemblies are now Signed
  • The new ConverterBase.DefaultDateTimeFormat to avoid setting the converter format field by field
  • Some error messages from the library were rewritten and now provide more context information (line and column numbers, field name, etc).
  • ExcelStorage support for more than 26 columns (thxs to Mark Izendooren)
  • Encoding Support to the CsvEngine
  • The wizard remembers all the paths now
  • SkipThisRecord in the AfterReadRecordEventArgs, this allows the skip records from the results (thanks Crestline)
  • More Methods for the CommonEngine
  • More than 420+ NUnit tests!!! (I can't belive it, I´m a lazy developer but I know how much worth there is in testing the lib)
  • Now you have two versions for the demos, one for each framework version.
  • The FixedLengthClassBuilder has more constructor to set the lengths of the fields in you instruction
  • Improved docs. (Thanks Antoine and Matt)
  • A lot of new examples check it out !!

Browse the complete history at SourceForge.

RoadMap (the future of the lib)

Check it out at SourceForge FileHelpers Roadmap.

Contact and Ideas

The main site of contact of the library are the FileHelpers Forums

Quick Feedback FileHelpers Forums

If you find that there is a feature that I must include, or you have a new idea (for the API, Source Code or Examples), only let me know, entering the forums or sending an e-mail to marcos[at]filehelpers.com

Licence (LGPL)

FileHelpers Library is @Copyright 2005-2006 to Marcos Meli but it's source code and the binaries are free for commercial and non commercial use.

Vote for this article if you like the library.

License

This article, along with any associated source code and files, is licensed under The BSD License

Share

About the Author

Marcos Meli
Software Developer (Senior)
Argentina Argentina
Marcos Meli v1.0 was released at august of 1980 by Francisco and Mabel Inc.
He´s from Bahia Blanca, Argentina.
 
He enjoy developing from the 12 years, has a degree in Computer Science, and of course, love this site and .NET in general.
 
He is the lead developer of the FileHelpers Library.
 
Marcos is also the co-funder of Devoo.Net.
A company that provides Object Oriented Components & Libraries for .NET developers (mostly open source)

Comments and Discussions

 
QuestionCSV to SQL with c# PinmemberHind Ahmamad4-Aug-14 2:10 
QuestionSupport for mysql? Pinmemberinvisible.symbol5-Sep-13 3:04 
AnswerRe: Support for mysql? PinmemberRob Warthen10-Mar-14 5:34 
QuestionVisual Studio 2003 and Compact Framework 1.1? PinmemberJaspar Lamar Crabb8-Aug-13 23:43 
QuestionHow to check a file is open or not in FileHelper PinmemberPraveen Jeganathan7-May-13 20:55 
QuestionProblem with errro handling simple files PinmemberBrian D Hutchison6-Feb-13 6:31 
GeneralMy vote of 5 PinmemberDevMostafa27-Dec-12 0:51 
QuestionSort text file by chars 5 to 100, big files (4GB) Pinmemberwasha7-Dec-12 7:55 
I need to sort large text files (+4GB).
Basically using a substring, something like:
lineInput.Substring(5, 100)
 
I've been reading the documentation and see that the library can sort files using records:
CommonEngine.SortFile
Public Shared Sub SortFile(ByVal recordClass As Type, _
   ByVal sourceFile As String, ByVal sortedFile As String)
 
But what about using a substring?
and handling 4GB files?
 
Thanks for any help.
AnswerRe: Sort text file by chars 5 to 100, big files (4GB) PinmemberMarcos Meli7-Dec-12 8:54 
GeneralRe: Sort text file by chars 5 to 100, big files (4GB) Pinmemberwasha16-Dec-12 11:17 
GeneralRe: Sort text file by chars 5 to 100, big files (4GB) PinmemberMarcos Meli16-Dec-12 15:47 
GeneralMy vote of 5 Pinmembermertg1-Apr-12 4:00 
QuestionThe directory is not empty. Pinmemberleeg12121-Feb-12 7:25 
AnswerRe: The directory is not empty. PinmemberJeff Firestone5-Mar-12 9:46 
GeneralLatest version PinmemberJeff Firestone21-Dec-11 9:22 
GeneralRe: Latest version PinmemberGary Noble6-Feb-12 1:52 
QuestionSeparador costumizado Pinmembermigc200413-Sep-11 4:33 
QuestionMy Vote of 5 PinmemberRaviRanjankr4-Sep-11 9:23 
QuestionProblem with Excel Pinmemberrainerdam8-Aug-11 12:09 
GeneralMy vote of 5 PinmemberlRUSHl21-Jun-11 0:26 
GeneralAdvise PinmemberMember 218608623-Mar-11 21:56 
GeneralRe: Advise PinmemberMarcos Meli31-Mar-11 6:42 
GeneralSome missing functionality Pinmemberbdaniel721-Mar-11 23:44 
QuestionI have a CSV file where the Date and the time are into 2 fields. How can I use FileHelpers to aggregate the 2 fields into the same DateTime data ? PinmemberTintin9216-Mar-11 13:34 
QuestionHeaderText PinmemberWbehning11-Mar-11 9:04 
QuestionCompare To LINQToCSV PinmemberPeter Lee1-Feb-11 7:37 
AnswerRe: Compare To LINQToCSV PinmemberPeter Lee1-Feb-11 7:39 
GeneralMy vote of 5 PinmemberSunasara Imdadhusen30-Dec-10 20:16 
GeneralMy vote of 5 Pinmembersellinger3-Dec-10 0:38 
GeneralError with 2.1 GB txt file Pinmemberchadok2-Dec-10 5:15 
GeneralRe: Error with 2.1 GB txt file PinmemberMarcos Meli2-Dec-10 6:02 
GeneralRe: Error with 2.1 GB txt file Pinmemberchadok2-Dec-10 6:13 
GeneralRe: Error with 2.1 GB txt file Pinmemberchadok2-Dec-10 6:42 
GeneralRe: Error with 2.1 GB txt file PinmemberMarcos Meli2-Dec-10 7:20 
GeneralRe: Error with 2.1 GB txt file Pinmemberchadok2-Dec-10 7:27 
GeneralRe: Error with 2.1 GB txt file PinmemberMarcos Meli2-Dec-10 7:57 
GeneralRe: Error with 2.1 GB txt file Pinmemberchadok2-Dec-10 14:41 
GeneralRe: Error with 2.1 GB txt file PinmemberMarcos Meli2-Dec-10 15:14 
GeneralRe: Error with 2.1 GB txt file PinmemberJeff Firestone21-Jan-11 9:36 
GeneralMy vote of 5 Pinmembersopheak.programming14-Nov-10 5:01 
GeneralMy vote of 5 PinmemberMember 453449511-Nov-10 7:19 
GeneralSmall problem with the links PinmemberOriginalGriff22-Oct-10 22:05 
Questionhow to convert number like "3.14E-5" to double value? Pinmembercjhren25-Sep-10 23:07 
GeneralMy vote of 5 PinmemberKPryor20-Sep-10 4:52 
GeneralMy vote of 5 Pinmemberfredchan52114-Sep-10 5:03 
GeneralHeader Record PinmemberAndylMurphy20-Jul-10 23:41 
GeneralDealing with Single Quotes Pinmemberenexooone13-May-10 1:40 
QuestionStep by Step Guide? Pinmemberenexooone29-Apr-10 22:51 
GeneralMy vote of 5 Pinmemberjarimba4-Feb-10 23:35 
GeneralGreat! PinmemberPolymorpher20-Jan-10 13:36 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140827.1 | Last Updated 10 Jul 2007
Article Copyright 2005 by Marcos Meli
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid