FileHelpers v2.0 - Delimited (CSV) or Fixed Data Import/Export Framework

• Download the binaries, demo, wizard and docs for .NET 1.1, 2.0, MONO and PPC - 3.200 Kb (local copy)
• Download the Sources for VS2003 and VS2005 - 3.250 Kb (local copy)

FileHelpers Home Page (Project Summary)

FileHelpers v 2.0

Introduction History and Last Changes !! Screencasts of the library Basic Uses Quick Start Guide Class Wizard Who Needs the FileHelper Library ? Main Features - Others Features Class Diagram RoadMap Level Extreme Article - Strong type your flat file with 2 lines of code

The FileHelpers are an easy to use library to import/export data from fixed length or delimited flat files (like the CSV). FileHelpers also has support to import/export data from different data storages (Excel, Acces, SqlServer).

The library has a set of converters for the basic types and can be easily extended to provide custom converters.

The main idea is pretty simple:

You can strong type your flat file (fixed or delimited) simply describing a class that maps to each record of the file and work with your file like a strong typed .NET array.

Directly between the files and the .NET source code:

It also can be used as an intermediate between files and MS Access or MS SqlServer tables:

To start using the FileHelpers Library you only need to add a reference in your project to the file: FileHelpers.dll. You can find it in the Release directory of the distribution. Tip: remember to leave the FileHelpers.xml file to get Intellisense support.

Next you need to define a class that maps to the record in the source/detination file. For this example we use a file with this format delimited by a |:

10248|VINET|04071996|32.38
10249|TOMSP|05071996|11.61
10250|HANAR|08071996|65.83
10251|VICTE|08071996|41.34
...............

The class that we refer can be like this:

[DelimitedRecord("|")]
public class Orders
{
   public int OrderID;

   public string CustomerID;

   [FieldConverter(ConverterKind.Date, "ddMMyyyy")]
   public DateTime OrderDate;

   public decimal Freight;
}

Later you must instantiate a FileHelperEngine and Read/Write files:

FileHelperEngine engine = new FileHelperEngine(typeof(Orders));

/// to Read use:
Orders[] res = (Orders[]) engine.ReadFile("input.txt");

/// to Write use:
engine.WriteFile("output.txt", res);

Finally you can use the res array to access each item in the file, for example:

 foreach (Orders order in res)
 {
    Console.WriteLine("Order Info:");
    Console.WriteLine(order.CustomerID + " - " +
                      order.OrderDate.ToString("dd/MM/yy"));
 }

Working with FixedLength files is exactly the same but you need to use the [FixedLengthRecord] and [FieldFixedLength] attributes.

Since version 1.3.5 the library has had a Record Class Wizard to generate the record class. The Record Class Wizard allows you to generate your record mapping class, save the definition, generate code based on snippets (Read File, Read with Generics, ReadAsync and so on).

The wizard uses the excellent FireBall CodeEditor

Click to see how it works

In almost every project there is a need to read/write data from/to a flat file of a specified format. Writing the code to process these files is not hard, you can use String.Split or String.SubString, but if you do that the code sometimes becomes hard to read and change. If you want to add support later for quoted string, multiline, convert types, etc. these things tend to get complicated.

So with the FileHelpers you don't need to worry about changes in the file structure because you can change the Type, Converters, Triming, Length in seconds.

Another place where you can find the FileHelpers useful is for log parsing, data warehouse and OLAP applications, communication between systems, file format transformations (for example from a fixed length to an CSV file), load test data into your NUnit tests and a lot more!!!

This library aims to provide an easy and reliable way to accomplish all these tasks.

Easy to use: The FileHelpers Lib is straight forward to learn and use. (see EasyExample)

Auto Converters: The library has a set of converters for the basic types and can be easy extended to provide custom converters. (see ConverterBase and Converter Example)

RunTime Classes since version 1.6.0 you have been able to create your record class at run time, and load them from files with source code or an XML description (check the example)

Master-Detail: You can read and write records with a master/detail pattern. (see Example1 and Example2)

Multiple record format support: With the MultirecordEngine you can read files with different record layout, you can also read files with some delimited and some fixed length records. (see Example)

Event Support: The engines of the library contain some events to allow you to easily extend the behavior of the library (see Example)

MS Excel Storage: Are a way to extract / insert records between any source and an excel file. (see ExcelDataStorage and ExcelDataLinkExample)

DataLinks: Are a way to extract / insert records between a database and a file. (see DataStorage and FileDataLinkExample)

GenericDataLink: Now you can to copy records between two Data Storages (see GenericDataLink and DataStorage)

Asynchronous Mode: You can use the library to read line by line and not the whole file. (see Async Methods)

.NET Compact Framework Support: From the version 1.1 you can use the FileHelpers library for you PocketPC and WindowsCE developments. (thanks Pierre)

File Transform Engine: To convert files in one format to another (for example a file with CSV to a FixedLength record format) (check the example)

Progress Notification: To get notified of the progress in each operation in the library (check the example)

.NET 2.0 Generics: the cast less and strong typed version of the engines (check the example)

FileDiffEngine to allow compare files with the same record layout (check the example)

Files, Stream and String Support: You can use the library to read/write any stream or string, not only files. (see FileHelperEngine)

Different Error Behaviors: You can set the behavior of the library when an error is found (throw and exception, ignore, save and continue, etc) (see the examples)

Quoted String Support: It allows to indicates that the field must be read and written as Quoted String, like Excel CSV. (see FieldQuotedAttribute )

CommonEngine a easy way to access to common operations (check the example)

Encoding Aware: You can define the encoding used to read and write files or streams. (see BaseEngine.Encoding)

NullValues: The library identifies the null values in the files and assigns an adequate value. (see FieldNullValueAttribute)

Good Documentation: The library is fully documented (at least that's the intention) with a lot of Examples of Use.

Align and Trimming: You can set the processing mode of the in/out strings with a lot of attributes like the FieldAlignAttribute and the FieldTrimAttribute.

Progress Notification

Here is a screenshot of the progress notification of the demo app.

Since versión 1.4.0 you can get notified of the progress of each operation of the library to show feedback to the user or whatever you want.

Detailed Diagram

Testing

The library contains more than 250 NUnit tests to ensure correctness:

The library has passed his 1st birthday!!! Thank you all for your work and contributions (partial list of them)

A lot of things were changed (mostly internally), now we have a performance gain of more than 60% for .NET 2.0

I keep releasing the .NET 1.1 version because I know that a lot of people use it in some corportaions, but it could be that new features in the versions to come will be .NET 2.0 only. It is monumental work to maintain and optimize both versions.

There are also a lot of major changes, a lot of refactoring in the code, and a big core rewrite that allows the engines to use less memory and temporary strings (we are using more buffers to aviod this)

I keep working hard to keep the library updated and give support to the users. You can check for examples in the Forums which shows a lot of problems that have been solved. I also ask for a lot of mail with problems from the users.

We have our own domain www.filehelpers.com thanks to (Antoine) and a Development Blog

Breaking Changes

• The constructor of the MultiRecordEngine was changed to allow params args, and to disallow users from passing the RecordSelector (in write operations were not needed).
• EndsRead and EndsWrite deleted, now you have a Close() operation to simplify the API and to make it similar to the System.Data namespace. (The async engines also implement IDisposable)
• Renamed FileHelperException --> FileHelpersException (a better name and an easy solution to a problem)
• By default the numeric fields in the FixedLengthRecords that don't have a [FieldAlign] will be aligned to the right and the rest the left by default, with this we avoid the problem of generating files with different meaning when we read and write.

Performance Related

• Internal use of Reflection.Emit, Dynamic Methods and char buffers to get more than a 50% enhace in performance and for .NET 2.0 more than 65%
• The Read..AsDT methods now create the DataTable record by record, and not at the end so you can handle large files without any memory overload
• Not more reflection in the operations (only in the constructors).
• FieldSorter faster too, Removed REFLECTION for EMIT

Shining new features

• The very cool DelimitedFileEngine and FixedFileEngine that allow to change options at RunTime (also with generic versions) Check an example here
• ConditionalRecords you can easily include or exclude certain records based on a RecordCondition (like BeginsWith, EndsWith, Contains or RegEx) Check the example
• A new GenericDatabaseStorage which is excellent for allowing others Db with ASO.NET support to work with the FileHelpers (Thanks Rodolfo Finochietti)
• Experimental Mono Project Support (some users are already using it, we hope that in the next release we have a completely working version for this framework)
• New Version Checker in the demos and Wizard (both with renew look and feel) Check the movies
• .NET 2.0 Nullable Types support. (Thanks Vijayan) Check the example
• Notification Interfaces: you can now get notified of the events simply implement INotifyRead or INotifyWrite
• Enhaced Debugging in .NET 2.0 with DebuggerDisplay and DebuggerBrowsable, DebugVisualizars to come.
• The Wizard has a Record Class Test that allows you to introduce sample data and your class and check for errors or results.
• You can check some screencast about the demos, wizard and other features of the library.
• Now the code for VS2003 and VS2005 are both on SVN, sharing the same files and the build script autodetect your Visual Studio installation. So no more excuses: time to get involved.
• The engines with async ops are now IDisposable and IEnumerable so you can declare it with using(..) and use them in a foreach loop

API Changes and Extensions

• The write methods are less strict. You can pass now an IEnumerable instead of an array, so you can direct pass a List or ArrayLIst to the method without needing to do a ToArray()
• RunTime records can now get a DataTable in the constructor and so use DataColumns to create one field for each column.
• Generic versions of near all the engines of the library (it's really hard to mantain the two copies so everyone, it's time to port!!)
• Async Operations in others engines like the MultiRecordEngine
• Event support to the MultiRecordEngine thanks to the contribution of Francis de Fouchier
• The library handle now has infinite levels of inheritance adding the high level fields first in the result record class.
• Allow to ignore spaces and tabs in the IgnoreEmptyLinesAttribute
• IgnoreCommentedLines thanks to MCampbell
• The Read methods now have a maxRecords args to tell to the engines how much they must read.
• A new constructor overload to the engines that allows you to pass the Encoding, which is useful for making the users aware of the Encoding feature.
• FileTransformEngine with two new methods ReadAndTransformRecords and TransformRecords
• Flush Method to AsyncEngines to allow users to ensure data write, for example, if they are using the lib for logging.
• The converters are now smarter. They validate to which types can be assigned and throw exceptions if it is wrong (for example if you use ConverterKind.Decimal in an int field)
• The ConvertException has a lot of context information: LineNumber, ColumnNumber, FieldName. Also has a better Exception Message.
• The integer converters now recieve a decimal separator to build an InvariantCulture to format and parse the values and strings

Small Changes

• Fixed the problem with ASP.NET: "Line 0: Metadata file 'filehelpers.dll' could not be found"
• Fixed some problems with the CsvClassBuilder (it's a good practice to not release features added last-minute)
• The Assemblies are now Signed
• The new ConverterBase.DefaultDateTimeFormat to avoid setting the converter format field by field
• Some error messages from the library were rewritten and now provide more context information (line and column numbers, field name, etc).
• ExcelStorage support for more than 26 columns (thxs to Mark Izendooren)
• Encoding Support to the CsvEngine
• The wizard remembers all the paths now
• SkipThisRecord in the AfterReadRecordEventArgs, this allows the skip records from the results (thanks Crestline)
• More Methods for the CommonEngine
• More than 420+ NUnit tests!!! (I can't belive it, I´m a lazy developer but I know how much worth there is in testing the lib)
• Now you have two versions for the demos, one for each framework version.
• The FixedLengthClassBuilder has more constructor to set the lengths of the fields in you instruction
• Improved docs. (Thanks Antoine and Matt)
• A lot of new examples check it out !!

Browse the complete history at SourceForge.

Check it out at SourceForge FileHelpers Roadmap.

Contact and Ideas

The main site of contact of the library are the FileHelpers Forums

If you find that there is a feature that I must include, or you have a new idea (for the API, Source Code or Examples), only let me know, entering the forums or sending an e-mail to marcos[at]filehelpers.com

Licence (LGPL)

FileHelpers Library is @Copyright 2005-2006 to Marcos Meli but it's source code and the binaries are free for commercial and non commercial use.

Vote for this article if you like the library.