Click here to Skip to main content
15,861,125 members
Articles / Web Development / HTML

FileHelpers v3.1 - Delimited (CSV) or Fixed Data Import/Export Framework

Rate me:
Please Sign up or sign in to vote.
4.84/5 (447 votes)
17 Aug 2015MIT8 min read 2.8M   2.8K   973   364
An easy to use .NET library to read/write strong typed data from files with fixed length or delimited records (CSV). Also has support to import/export data from different data storages (Excel, Acces, SqlServer, MySql)

FileHelpers Home Page

FileHelpers v 3.1

Image 1

Introduction

The FileHelpers are an easy to use library to import/export data from fixed length or delimited flat files (like the CSV). FileHelpers also has support to import/export data from different data storages (Excel, Acces, SqlServer).

The library has a set of converters for the basic types and can be easily extended to provide custom converters.

The main idea is pretty simple:

You can strong type your flat file (fixed or delimited) simply describing a class that maps to each record of the file and work with your file like a strong typed .NET array.

Image 2

Basic Uses

Directly between the files and the .NET source code:

general read write of filehelpers 

It also can be used as an intermediate between files and MS Access or MS SqlServer tables:

 

General read write to data storage

Download Instructions

The recommend method to install the library is use the NuGet Package

www.nuget.org/packages/FileHelpers/

Via de NuGet Console:     Install-Package FileHelpers

Or you can use the Visual Studio NuGet Explorer to search for FileHelpers

add references

 

Quick Start Guide (easy Image 6Image 7Image 8Image 9 steps)

Image 10

To start using the FileHelpers Library you only need to add a reference in your project to the file: FileHelpers.dll. You can find it in the Release directory of the distribution. Tip: remember to leave the FileHelpers.xml file to get Intellisense support.

Image 11

Image 12

Next you need to define a class that maps to the record in the source/detination file. For this example we use a file with this format delimited by a |:

10248|VINET|04071996|32.38
10249|TOMSP|05071996|11.61
10250|HANAR|08071996|65.83
10251|VICTE|08071996|41.34
...............

The class that we refer can be like this:

C#
[DelimitedRecord("|")]
public class Orders
{
   public int OrderID;

   public string CustomerID;

   [FieldConverter(ConverterKind.Date, "ddMMyyyy")]
   public DateTime OrderDate;

   public decimal Freight;
}

Image 13

Later you must instantiate a FileHelperEngine and Read/Write files:

C#
var engine = new FileHelperEngine<Orders>();

/// to Read use:
var res = engine.ReadFile("input.txt");

/// to Write use:
engine.WriteFile("output.txt", res);

Image 14

Finally you can use the res array to access each item in the file, for example:

C#
foreach (Orders order in res)
{
   Console.WriteLine("Order Info:");
   Console.WriteLine(order.CustomerID + " - " +
                     order.OrderDate.ToString("dd/MM/yy"));
}

Working with FixedLength files is exactly the same but you need to use the [FixedLengthRecord] and [FieldFixedLength] attributes.

More Examples

We have ton of examples in our web site to learn to use the library from every angle

Image 15

Who needs the File Helpers Library?

In almost every project there is a need to read/write data from/to a flat file of a specified format. Writing the code to process these files is not hard, you can use String.Split or String.SubString, but if you do that the code sometimes becomes hard to read and change. If you want to add support later for quoted string, multiline, convert types, etc. these things tend to get complicated.

So with the FileHelpers you don't need to worry about changes in the file structure because you can change the Type, Converters, Triming, Length in seconds.

Another place where you can find the FileHelpers useful is for log parsing, data warehouse and OLAP applications, communication between systems, file format transformations (for example from a fixed length to an CSV file), load test data into your NUnit tests and a lot more!!!

This library aims to provide an easy and reliable way to accomplish all these tasks.

Roslyn Analyzer

Our Roslyn Analyzer helps you to use the library in the right way.

Image 16

The current Analyzer implements:

  • Recomends to use the generic version when you use an engine with a typeof() in the constructor
  • Suggest to use [FieldHidden] instead of [FieldIgnored] or [FieldNotInFile]
  • Use FileHelpers.Dynamic instead of FileHelpers.RunTime namespace
  • Use IComparable instead of the obsolete IComparableRecord 

next version will implement

  • Check that record class contains valid record attributes
  • Check that record class contains any field
  • Check other obsolete methods
  • Convert engine.ReadFile in async version with foreach

You can join the development of the analyzer, join us in the Gitter chat:

https://gitter.im/MarcosMeli/FileHelpers

 

 

Record Class Wizard

Since version 1.3.5 the library has had a Record Class Wizard to generate the record class. The Record Class Wizard allows you to generate your record mapping class, save the definition, generate code based on snippets (Read File, Read with Generics, ReadAsync and so on).

Image 17
Click to see how it works

Main Features

 

 Easy to use

The FileHelpers library is straightforward to learn and use.
Check the Examples

 Auto Converters

The library has a set of converters for common types and can be easy extended to provide custom converters.
Check the Example

 Event Support

The engines of the library contain events to make it easier to extend the behavior of the library
Check the INotifyRead andINotifyWrite Examples

 High Performance

The library doesn't use reflection to get or set the field values, it uses dynamic code generation

 Wide framework support

You can use the library in
.NET 2.03.04.04.54.6 and Mono!!

 Multiple Error Modes

You can configure the library to use different ErrorModes.
Check the example

 BigFileSorter

You can use the library to sort files with millon of records and MBs
Check the Example

 File Transform Engine

To convert files in one format to another (for example a file with CSV to a fixed length record format)
Check the Example

 FileDiffEngine

Provides the ability to compare files with the same record layout
Check the Example

 MasterDetailEngine

You can process hierarchical data with this engine
Check the Example

 Nullable Types

The library supports Nullable types in the core
Check the Example

 Record by Record Mode

You can use the library to read line by line and not the whole file.
Check the Example

 Open Source

The library is completely free to use in any kind of development
github.com/MarcosMeli/FileHelpers

 Format Autodetection

The library has an smart feature that based in some sample files can deduce the Record Class
Check the Example

 Wizard

We provide a Wizard app to help in the creation of the Record Class
Check the Online Wizard
 

 

Progress Notification

Here is a screenshot of the progress notification of the demo app.

Since versión 1.4.0 you can get notified of the progress of each operation of the library to show feedback to the user or whatever you want.

Image 18

Class Diagram

Image 19

Detailed Diagram

Detailed Diagram

Testing

The library contains more than 650 NUnit tests to ensure correctness:

Sample image

 
List of Converters Arguments

Here is a list of the parameters that you can give to the default converters:

ConverterKind.Date

Arg1: A string with the DateTime format that the engine passes to the DateTime.Parse function.

Arg2: A string with the encoding used for string convertions

Examples:

C#
// By default the engines use: "ddMMyyyy"
[FieldConverter(ConverterKind.Date)]
public DateTime ShippedDate;

// Parse these dates:  1-1-2006   01-1-2006   30-01-2006
[FieldConverter(ConverterKind.Date, "d-M-yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  1-1-2006   01-1-2006   01-30-2006
[FieldConverter(ConverterKind.Date, "M-d-yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  01/3/2006   30/02/2006
[FieldConverter(ConverterKind.Date, "d/M/yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  01042006   30022006
[FieldConverter(ConverterKind.Date, "ddMMyyyy" )]
public DateTime ShippedDate;

// Parse these dates:  04/January/2006   02/July/2006
[FieldConverter(ConverterKind.Date, "dd/MMMM/yyyy", "en" )] // or "en-US"
public DateTime ShippedDate;

// Parse these dates:  04/Enero/2006   02/Julio/2006
[FieldConverter(ConverterKind.Date, "dd/MMMM/yyyy", "es" )]
public DateTime ShippedDate;

You can check all the supported format strings check the MSDN docs for DateTime.ParseExact 
and here all support cultures

ConverterKind.Double, ConverterKind.Single and ConverterKind.Decimal

Arg1: A string with the character to be used as DecimalSeparator. Valid Values: "." or ",". By default: "."

Examples:

C#
// "." is the decimal separator by default
[FieldConverter(ConverterKind.Double)]
public double Freight;

// "." is the decimal separator. (good for autodocumented code)
[FieldConverter(ConverterKind.Double, ".")]
public double Freight;

// "," is the decimal separator.
[FieldConverter(ConverterKind.Double, ",")]
public double Freight;

// The same for the other converters:
// ConverterKind.Decimal and ConverterKind.Single

Integer Converters

ConverterKind.Int16, ConverterKind.Int32, ConverterKind.Int64, ConverterKind.Byte, 
ConverterKind.UInt16, ConverterKind.UInt32, ConverterKind.UInt64, and ConverterKind.SByte

Arg1: A string with the character to be used as DecimalSeparator. Valid Values: "." or ",". By default: "."

WARNING: The library requires a decimal separator here and internally creates the group separator with the counterpart (for example if you provide "." it uses ",")

Examples:

C#
// "." is the decimal separator by default
// allows you to parse:  12,125 or 1,458,385
[FieldConverter(ConverterKind.Int32)]
public Int32 Amount;

// using "," as decimal separator
// allows you to parse:  12.125 or 1.458.385
[FieldConverter(ConverterKind.Int32, ",")]
public Int32 Amount;

// The same works for the rest of the converts

ConverterKind.Boolean

Arg1: A string that represents the True value 
Arg2: A string that represents the False value

By default, this converter takes the strings "True" (case insensitive) and "1" as true. The values "True" and "False" are returned when converting the field to string.

Examples:

C#
// By default it reads as true the strings "True" (Case insensitive) and "1"
[FieldConverter(ConverterKind.Boolean)]
public bool Shipped;

// Takes as true the string "verdad" and as false the string "mentira" (case insensitive)
[FieldConverter(ConverterKind.Boolean, "Verdad", "MenTiRa")]
public bool Shipped;

// Takes as true the string "1" and as false the string "0"
// You need these params if you want to write, because it use "True" and "False"
[FieldConverter(ConverterKind.Boolean, "1", "0")]
public bool Shipped;

// Takes as true the string "X" and as false the string "-"
[FieldConverter(ConverterKind.Boolean, "X", "-")]
public bool Shipped;

History

Version 3.1, July 2015

Hi everyone again !!

The FileHelpers has reached a big Milestone today we are releasing the 3.1 version!

Now with full support for .NET 4.0, 4.5 and Mono and with a ton of news and enhancements.

After some time off for different reasons we are now trying to update the library in a more regular fashion. We redesigned the HomeSite with a Material Design Template from GeeksLabs

Main changes

  • Docs rewritten thanks to the work of Ken Foskey and Matthias Cavigelli
  • NEW: Full .NET 2.0, 4.0 & 4.5 support.
  • NEW: Added support for MONO !!
  • NEW: FieldOrder: a new attribute to override the default order of fields.
  • NEW: Partial support to use Autoproperties
  • NEW: Async Methods returns IDisposable to allow using() statements that autoclose files on ends or exception.
  • NEW: Events for the Async engines.
  • NEW: The events args now have TotalBytes and CurrentBytes to allow you show progress while reading or writing.
  • NEW: FieldIndexers you can now get the values in the AsyncEngine like in the DataReader of ADO.NET =) You can access them via index or fieldName (at the moment case sensitive). This also completes the RunTime records support
  • NEW: SmartFormatDetector: A class designed to discover the format of delimited and flat files based on sample files. It is integrated with the wizzard
  • NEW: Better Delimited validations to make sure that the file contains the right number of fields
  • NEW: FieldValueDiscarted: a new attribute to allow the value of a particular field to be read but not stored in the record class (this was a very requested feature)
  • NEW: BigFileSorter: if you need to sort big files, you can use this feature that implements External Sorting
  • MOD: The error messages have been made more meaningful. In all exceptions you get: FieldName, ColumnNumber, LineNumber etc.
  • NEW: FieldNotEmpty: Indicates whether the target field needs to be populated with a non-empty value.

Breaking Changes

  • Removed support for .NET Compact Framework
  • [FieldIgnored] => [FieldHidden]
  • IComparableRecord<T> => IComparable<T>

Minor Changes

  • Better caching of internal classes
  • The core code of the library was refactored to make easier to extend it.
  • Better performance.
  • ADD: UpdateLinks properties, allow to specify how the library must handle Workbook links (thanks to Stefan Schälle)
  • A ton of new examples, autogenerated from FileHelpers.Examples project
  • FIX: A little problem with the quoted fields in the wizard
  • More documentation has been added to the Library source code. It is now even easier to get involved in development!
  •  
 

Licence (MIT)

FileHelpers Library is @Copyright 2005-2015 to Marcos Meli but it's source code and the binaries are free for commercial and non commercial use.

Vote for this article if you like the library.

License

This article, along with any associated source code and files, is licensed under The MIT License


Written By
Architect Devoo
Argentina Argentina
Marcos Meli v1.0 was released at august of 1980 by Francisco and Mabel Inc.
He´s from Bahia Blanca, Argentina.

He enjoy developing from the 12 years, has a degree in Computer Science, and of course, love this site and .NET in general.

He is the lead developer of the FileHelpers Library.

Marcos is also the co-funder of Devoo.Net.
A company that provides Object Oriented Components & Libraries for .NET developers (mostly open source)

Comments and Discussions

 
GeneralRe: Nullable Data Types Pin
Marcos Meli1-Apr-07 19:33
Marcos Meli1-Apr-07 19:33 
QuestionSlow when reading Excel Pin
chai9332313-Feb-07 15:34
chai9332313-Feb-07 15:34 
GeneralOld format or invalid type library. (Exception) Pin
DoMage12-Feb-07 21:05
DoMage12-Feb-07 21:05 
GeneralExport/Import Image data to (.txt) or (.xls) Pin
utehn13-Dec-06 20:10
utehn13-Dec-06 20:10 
GeneralRe: Export/Import Image data to (.txt) or (.xls) Pin
Marcos Meli18-Dec-06 5:12
Marcos Meli18-Dec-06 5:12 
GeneralRe: Export/Import Image data to (.txt) or (.xls) Pin
utehn18-Dec-06 19:31
utehn18-Dec-06 19:31 
QuestionRuntime class Pin
stefan stoian2-Oct-06 8:27
stefan stoian2-Oct-06 8:27 
AnswerRe: Runtime class Pin
Marcos Meli2-Oct-06 8:35
Marcos Meli2-Oct-06 8:35 
Hi Stefan

Because you are using VB.NET you need to pass that language to:

ClassBuilder.ClassFromSourceFile("c:\work\TestRuntime\TestRuntime\stef.vb", NetLanguage.VbNet)

Cheers
Marcos



QuestionIs it possible to export to Excel from Ms SQL Server? Pin
_Thurein_4-Sep-06 18:37
_Thurein_4-Sep-06 18:37 
AnswerRe: Is it possible to export to Excel from Ms SQL Server? Pin
Marcos Meli5-Sep-06 5:21
Marcos Meli5-Sep-06 5:21 
QuestionExcel Request Pin
Gary Noble23-Aug-06 6:55
Gary Noble23-Aug-06 6:55 
AnswerRe: Excel Request Pin
Marcos Meli23-Aug-06 7:12
Marcos Meli23-Aug-06 7:12 
GeneralCannot find the FileHelper File Pin
kylai219-Aug-06 23:50
kylai219-Aug-06 23:50 
GeneralRe: Cannot find the FileHelper File Pin
Marcos Meli20-Aug-06 4:16
Marcos Meli20-Aug-06 4:16 
GeneralRe: Cannot find the FileHelper File [modified] Pin
kylai220-Aug-06 15:37
kylai220-Aug-06 15:37 
GeneralRe: Cannot find the FileHelper File Pin
Marcos Meli21-Aug-06 16:33
Marcos Meli21-Aug-06 16:33 
GeneralRe: Cannot find the FileHelper File Pin
naiba1919-Sep-06 1:17
naiba1919-Sep-06 1:17 
GeneralRe: Cannot find the FileHelper File Pin
Marcos Meli19-Sep-06 3:28
Marcos Meli19-Sep-06 3:28 
GeneralRe: Cannot find the FileHelper File Pin
naiba1919-Sep-06 5:36
naiba1919-Sep-06 5:36 
GeneralRe: Cannot find the FileHelper File Pin
Marcos Meli19-Sep-06 7:18
Marcos Meli19-Sep-06 7:18 
GeneralRe: Cannot find the FileHelper File Pin
roumenf4-Dec-06 0:41
roumenf4-Dec-06 0:41 
GeneralRe: Cannot find the FileHelper File Pin
Marcos Meli4-Dec-06 2:33
Marcos Meli4-Dec-06 2:33 
GeneralCongratulations on v1.6 Pin
Antoine Marie17-Aug-06 5:26
Antoine Marie17-Aug-06 5:26 
GeneralRe: Congratulations on v1.6 Pin
Marcos Meli17-Aug-06 15:47
Marcos Meli17-Aug-06 15:47 
GeneralRe: Congratulations on v1.6 Pin
vthg2themax16-Oct-06 8:02
vthg2themax16-Oct-06 8:02 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.