Click here to Skip to main content
11,705,473 members (44,150 online)
Click here to Skip to main content

FileHelpers v3.1 - Delimited (CSV) or Fixed Data Import/Export Framework

, 17 Aug 2015 MIT 1.4M 2.3K 955
Rate this:
Please Sign up or sign in to vote.
An easy to use .NET library to read/write strong typed data from files with fixed length or delimited records (CSV). Also has support to import/export data from different data storages (Excel, Acces, SqlServer, MySql)

FileHelpers Home Page

FileHelpers v 3.1

Introduction

The FileHelpers are an easy to use library to import/export data from fixed length or delimited flat files (like the CSV). FileHelpers also has support to import/export data from different data storages (Excel, Acces, SqlServer).

The library has a set of converters for the basic types and can be easily extended to provide custom converters.

The main idea is pretty simple:

You can strong type your flat file (fixed or delimited) simply describing a class that maps to each record of the file and work with your file like a strong typed .NET array.

Basic Uses

Directly between the files and the .NET source code:

general read write of filehelpers 

It also can be used as an intermediate between files and MS Access or MS SqlServer tables:

 

General read write to data storage

Download Instructions

The recommend method to install the library is use the NuGet Package

www.nuget.org/packages/FileHelpers/

Via de NuGet Console:     Install-Package FileHelpers

Or you can use the Visual Studio NuGet Explorer to search for FileHelpers

add references

 

Quick Start Guide (easy steps)

To start using the FileHelpers Library you only need to add a reference in your project to the file: FileHelpers.dll. You can find it in the Release directory of the distribution. Tip: remember to leave the FileHelpers.xml file to get Intellisense support.

Next you need to define a class that maps to the record in the source/detination file. For this example we use a file with this format delimited by a |:

10248|VINET|04071996|32.38
10249|TOMSP|05071996|11.61
10250|HANAR|08071996|65.83
10251|VICTE|08071996|41.34
...............

The class that we refer can be like this:

[DelimitedRecord("|")]
public class Orders
{
   public int OrderID;

   public string CustomerID;

   [FieldConverter(ConverterKind.Date, "ddMMyyyy")]
   public DateTime OrderDate;

   public decimal Freight;
}

Later you must instantiate a FileHelperEngine and Read/Write files:

var engine = new FileHelperEngine<Orders>();

/// to Read use:
var res = engine.ReadFile("input.txt");

/// to Write use:
engine.WriteFile("output.txt", res);

Finally you can use the res array to access each item in the file, for example:

foreach (Orders order in res)
{
   Console.WriteLine("Order Info:");
   Console.WriteLine(order.CustomerID + " - " +
                     order.OrderDate.ToString("dd/MM/yy"));
}

Working with FixedLength files is exactly the same but you need to use the [FixedLengthRecord] and [FieldFixedLength] attributes.

More Examples

We have ton of examples in our web site to learn to use the library from every angle

Who needs the File Helpers Library?

In almost every project there is a need to read/write data from/to a flat file of a specified format. Writing the code to process these files is not hard, you can use String.Split or String.SubString, but if you do that the code sometimes becomes hard to read and change. If you want to add support later for quoted string, multiline, convert types, etc. these things tend to get complicated.

So with the FileHelpers you don't need to worry about changes in the file structure because you can change the Type, Converters, Triming, Length in seconds.

Another place where you can find the FileHelpers useful is for log parsing, data warehouse and OLAP applications, communication between systems, file format transformations (for example from a fixed length to an CSV file), load test data into your NUnit tests and a lot more!!!

This library aims to provide an easy and reliable way to accomplish all these tasks.

Roslyn Analyzer

Our Roslyn Analyzer helps you to use the library in the right way.

The current Analyzer implements:

  • Recomends to use the generic version when you use an engine with a typeof() in the constructor
  • Suggest to use [FieldHidden] instead of [FieldIgnored] or [FieldNotInFile]
  • Use FileHelpers.Dynamic instead of FileHelpers.RunTime namespace
  • Use IComparable instead of the obsolete IComparableRecord 

next version will implement

  • Check that record class contains valid record attributes
  • Check that record class contains any field
  • Check other obsolete methods
  • Convert engine.ReadFile in async version with foreach

You can join the development of the analyzer, join us in the Gitter chat:

https://gitter.im/MarcosMeli/FileHelpers

 

 

Record Class Wizard

Since version 1.3.5 the library has had a Record Class Wizard to generate the record class. The Record Class Wizard allows you to generate your record mapping class, save the definition, generate code based on snippets (Read File, Read with Generics, ReadAsync and so on).


Click to see how it works

Main Features

 

 Easy to use

The FileHelpers library is straightforward to learn and use.
Check the Examples

 Auto Converters

The library has a set of converters for common types and can be easy extended to provide custom converters.
Check the Example

 Event Support

The engines of the library contain events to make it easier to extend the behavior of the library
Check the INotifyRead andINotifyWrite Examples

 High Performance

The library doesn't use reflection to get or set the field values, it uses dynamic code generation

 Wide framework support

You can use the library in
.NET 2.03.04.04.54.6 and Mono!!

 Multiple Error Modes

You can configure the library to use different ErrorModes.
Check the example

 BigFileSorter

You can use the library to sort files with millon of records and MBs
Check the Example

 File Transform Engine

To convert files in one format to another (for example a file with CSV to a fixed length record format)
Check the Example

 FileDiffEngine

Provides the ability to compare files with the same record layout
Check the Example

 MasterDetailEngine

You can process hierarchical data with this engine
Check the Example

 Nullable Types

The library supports Nullable types in the core
Check the Example

 Record by Record Mode

You can use the library to read line by line and not the whole file.
Check the Example

 Open Source

The library is completely free to use in any kind of development
github.com/MarcosMeli/FileHelpers

 Format Autodetection

The library has an smart feature that based in some sample files can deduce the Record Class
Check the Example

 Wizard

We provide a Wizard app to help in the creation of the Record Class
Check the Online Wizard
 

 

Progress Notification

Here is a screenshot of the progress notification of the demo app.

Since versión 1.4.0 you can get notified of the progress of each operation of the library to show feedback to the user or whatever you want.

Class Diagram

Detailed Diagram

Detailed Diagram

Testing

The library contains more than 650 NUnit tests to ensure correctness:

Sample image

 
List of Converters Arguments

Here is a list of the parameters that you can give to the default converters:

ConverterKind.Date

Arg1: A string with the DateTime format that the engine passes to the DateTime.Parse function.

Arg2: A string with the encoding used for string convertions

Examples:

// By default the engines use: "ddMMyyyy"
[FieldConverter(ConverterKind.Date)]
public DateTime ShippedDate;

// Parse these dates:  1-1-2006   01-1-2006   30-01-2006
[FieldConverter(ConverterKind.Date, "d-M-yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  1-1-2006   01-1-2006   01-30-2006
[FieldConverter(ConverterKind.Date, "M-d-yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  01/3/2006   30/02/2006
[FieldConverter(ConverterKind.Date, "d/M/yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  01042006   30022006
[FieldConverter(ConverterKind.Date, "ddMMyyyy" )]
public DateTime ShippedDate;

// Parse these dates:  04/January/2006   02/July/2006
[FieldConverter(ConverterKind.Date, "dd/MMMM/yyyy", "en" )] // or "en-US"
public DateTime ShippedDate;

// Parse these dates:  04/Enero/2006   02/Julio/2006
[FieldConverter(ConverterKind.Date, "dd/MMMM/yyyy", "es" )]
public DateTime ShippedDate;

You can check all the supported format strings check the MSDN docs for DateTime.ParseExact 
and here all support cultures

ConverterKind.Double, ConverterKind.Single and ConverterKind.Decimal

Arg1: A string with the character to be used as DecimalSeparator. Valid Values: "." or ",". By default: "."

Examples:

// "." is the decimal separator by default
[FieldConverter(ConverterKind.Double)]
public double Freight;

// "." is the decimal separator. (good for autodocumented code)
[FieldConverter(ConverterKind.Double, ".")]
public double Freight;

// "," is the decimal separator.
[FieldConverter(ConverterKind.Double, ",")]
public double Freight;

// The same for the other converters:
// ConverterKind.Decimal and ConverterKind.Single

Integer Converters

ConverterKind.Int16, ConverterKind.Int32, ConverterKind.Int64, ConverterKind.Byte, 
ConverterKind.UInt16, ConverterKind.UInt32, ConverterKind.UInt64, and ConverterKind.SByte

Arg1: A string with the character to be used as DecimalSeparator. Valid Values: "." or ",". By default: "."

WARNING: The library requires a decimal separator here and internally creates the group separator with the counterpart (for example if you provide "." it uses ",")

Examples:

// "." is the decimal separator by default
// allows you to parse:  12,125 or 1,458,385
[FieldConverter(ConverterKind.Int32)]
public Int32 Amount;

// using "," as decimal separator
// allows you to parse:  12.125 or 1.458.385
[FieldConverter(ConverterKind.Int32, ",")]
public Int32 Amount;

// The same works for the rest of the converts

ConverterKind.Boolean

Arg1: A string that represents the True value 
Arg2: A string that represents the False value

By default, this converter takes the strings "True" (case insensitive) and "1" as true. The values "True" and "False" are returned when converting the field to string.

Examples:

// By default it reads as true the strings "True" (Case insensitive) and "1"
[FieldConverter(ConverterKind.Boolean)]
public bool Shipped;

// Takes as true the string "verdad" and as false the string "mentira" (case insensitive)
[FieldConverter(ConverterKind.Boolean, "Verdad", "MenTiRa")]
public bool Shipped;

// Takes as true the string "1" and as false the string "0"
// You need these params if you want to write, because it use "True" and "False"
[FieldConverter(ConverterKind.Boolean, "1", "0")]
public bool Shipped;

// Takes as true the string "X" and as false the string "-"
[FieldConverter(ConverterKind.Boolean, "X", "-")]
public bool Shipped;

History

Version 3.1, July 2015

Hi everyone again !!

The FileHelpers has reached a big Milestone today we are releasing the 3.1 version!

Now with full support for .NET 4.0, 4.5 and Mono and with a ton of news and enhancements.

After some time off for different reasons we are now trying to update the library in a more regular fashion. We redesigned the HomeSite with a Material Design Template from GeeksLabs

Main changes

  • Docs rewritten thanks to the work of Ken Foskey and Matthias Cavigelli
  • NEW: Full .NET 2.0, 4.0 & 4.5 support.
  • NEW: Added support for MONO !!
  • NEW: FieldOrder: a new attribute to override the default order of fields.
  • NEW: Partial support to use Autoproperties
  • NEW: Async Methods returns IDisposable to allow using() statements that autoclose files on ends or exception.
  • NEW: Events for the Async engines.
  • NEW: The events args now have TotalBytes and CurrentBytes to allow you show progress while reading or writing.
  • NEW: FieldIndexers you can now get the values in the AsyncEngine like in the DataReader of ADO.NET =) You can access them via index or fieldName (at the moment case sensitive). This also completes the RunTime records support
  • NEW: SmartFormatDetector: A class designed to discover the format of delimited and flat files based on sample files. It is integrated with the wizzard
  • NEW: Better Delimited validations to make sure that the file contains the right number of fields
  • NEW: FieldValueDiscarted: a new attribute to allow the value of a particular field to be read but not stored in the record class (this was a very requested feature)
  • NEW: BigFileSorter: if you need to sort big files, you can use this feature that implements External Sorting
  • MOD: The error messages have been made more meaningful. In all exceptions you get: FieldName, ColumnNumber, LineNumber etc.
  • NEW: FieldNotEmpty: Indicates whether the target field needs to be populated with a non-empty value.

Breaking Changes

  • Removed support for .NET Compact Framework
  • [FieldIgnored] => [FieldHidden]
  • IComparableRecord<T> => IComparable<T>

Minor Changes

  • Better caching of internal classes
  • The core code of the library was refactored to make easier to extend it.
  • Better performance.
  • ADD: UpdateLinks properties, allow to specify how the library must handle Workbook links (thanks to Stefan Schälle)
  • A ton of new examples, autogenerated from FileHelpers.Examples project
  • FIX: A little problem with the quoted fields in the wizard
  • More documentation has been added to the Library source code. It is now even easier to get involved in development!
  •  
 

Licence (MIT)

FileHelpers Library is @Copyright 2005-2015 to Marcos Meli but it's source code and the binaries are free for commercial and non commercial use.

Vote for this article if you like the library.

License

This article, along with any associated source code and files, is licensed under The MIT License

Share

About the Author

Marcos Meli
Architect Devoo
Argentina Argentina
Marcos Meli v1.0 was released at august of 1980 by Francisco and Mabel Inc.
He´s from Bahia Blanca, Argentina.

He enjoy developing from the 12 years, has a degree in Computer Science, and of course, love this site and .NET in general.

He is the lead developer of the FileHelpers Library.

Marcos is also the co-funder of Devoo.Net.
A company that provides Object Oriented Components & Libraries for .NET developers (mostly open source)

You may also be interested in...

Comments and Discussions

 
QuestionImport/export data from Excel Pin
Alanlamuk18-Aug-15 10:02
professionalAlanlamuk18-Aug-15 10:02 
QuestionHow does performance comapare against other libraries? Pin
tolsen6411-Aug-15 10:57
membertolsen6411-Aug-15 10:57 
AnswerRe: How does performance comapare against other libraries? Pin
Marcos Meli12-Aug-15 6:44
memberMarcos Meli12-Aug-15 6:44 
QuestionNow lauched as commercial? Pin
sreeyush sudhakaran9-Aug-15 18:50
professionalsreeyush sudhakaran9-Aug-15 18:50 
AnswerRe: It is free and open source Pin
Marcos Meli10-Aug-15 1:41
memberMarcos Meli10-Aug-15 1:41 
GeneralRe: It is free and open source Pin
sreeyush sudhakaran10-Aug-15 2:01
professionalsreeyush sudhakaran10-Aug-15 2:01 
Generalsupport for SQLite? Pin
Southmountain8-Aug-15 9:41
memberSouthmountain8-Aug-15 9:41 
GeneralRe: support for SQLite? Pin
Marcos Meli9-Aug-15 10:12
memberMarcos Meli9-Aug-15 10:12 
GeneralRe: support for SQLite? Pin
Southmountain9-Aug-15 14:35
memberSouthmountain9-Aug-15 14:35 
GeneralRe: support for SQLite? Pin
Marcos Meli11-Aug-15 8:45
memberMarcos Meli11-Aug-15 8:45 
GeneralRe: support for SQLite? Pin
Southmountain17-Aug-15 14:03
memberSouthmountain17-Aug-15 14:03 
QuestionMy vote of 5 Pin
Martin Hart Turner6-Aug-15 20:40
memberMartin Hart Turner6-Aug-15 20:40 
AnswerRe: My vote of 5 Pin
Marcos Meli7-Aug-15 3:13
memberMarcos Meli7-Aug-15 3:13 
QuestionLeer archivos de mas de 5 gb Pin
Member 26943073-Nov-14 10:07
memberMember 26943073-Nov-14 10:07 
AnswerRe: Leer archivos de mas de 5 gb Pin
Marcos Meli7-Aug-15 8:22
memberMarcos Meli7-Aug-15 8:22 
QuestionCSV to SQL with c# Pin
Hind Ahmamad4-Aug-14 2:10
memberHind Ahmamad4-Aug-14 2:10 
AnswerRe: CSV to SQL with c# Pin
Mehulio Shah21-Oct-14 4:20
memberMehulio Shah21-Oct-14 4:20 
QuestionSupport for mysql? Pin
invisible.symbol5-Sep-13 3:04
memberinvisible.symbol5-Sep-13 3:04 
AnswerRe: Support for mysql? Pin
Rob Warthen10-Mar-14 5:34
memberRob Warthen10-Mar-14 5:34 
QuestionVisual Studio 2003 and Compact Framework 1.1? Pin
Jaspar Lamar Crabb8-Aug-13 23:43
memberJaspar Lamar Crabb8-Aug-13 23:43 
QuestionHow to check a file is open or not in FileHelper Pin
Praveen Jeganathan7-May-13 20:55
memberPraveen Jeganathan7-May-13 20:55 
QuestionProblem with errro handling simple files Pin
Brian D Hutchison6-Feb-13 6:31
memberBrian D Hutchison6-Feb-13 6:31 
GeneralMy vote of 5 Pin
DevMostafa27-Dec-12 0:51
memberDevMostafa27-Dec-12 0:51 
QuestionSort text file by chars 5 to 100, big files (4GB) Pin
washa7-Dec-12 7:55
memberwasha7-Dec-12 7:55 
AnswerRe: Sort text file by chars 5 to 100, big files (4GB) Pin
Marcos Meli7-Dec-12 8:54
memberMarcos Meli7-Dec-12 8:54 
GeneralRe: Sort text file by chars 5 to 100, big files (4GB) Pin
washa16-Dec-12 11:17
memberwasha16-Dec-12 11:17 
GeneralRe: Sort text file by chars 5 to 100, big files (4GB) Pin
Marcos Meli16-Dec-12 15:47
memberMarcos Meli16-Dec-12 15:47 
GeneralMy vote of 5 Pin
mertg1-Apr-12 4:00
membermertg1-Apr-12 4:00 
QuestionThe directory is not empty. Pin
leeg12121-Feb-12 7:25
memberleeg12121-Feb-12 7:25 
AnswerRe: The directory is not empty. Pin
Jeff Firestone5-Mar-12 9:46
memberJeff Firestone5-Mar-12 9:46 
GeneralLatest version Pin
Jeff Firestone21-Dec-11 9:22
memberJeff Firestone21-Dec-11 9:22 
GeneralRe: Latest version Pin
Gary Noble6-Feb-12 1:52
memberGary Noble6-Feb-12 1:52 
QuestionSeparador costumizado Pin
migc200413-Sep-11 4:33
membermigc200413-Sep-11 4:33 
QuestionMy Vote of 5 Pin
RaviRanjankr4-Sep-11 9:23
memberRaviRanjankr4-Sep-11 9:23 
QuestionProblem with Excel Pin
rainerdam8-Aug-11 12:09
memberrainerdam8-Aug-11 12:09 
GeneralMy vote of 5 Pin
lRUSHl21-Jun-11 0:26
memberlRUSHl21-Jun-11 0:26 
GeneralAdvise Pin
Member 218608623-Mar-11 21:56
memberMember 218608623-Mar-11 21:56 
GeneralRe: Advise Pin
Marcos Meli31-Mar-11 6:42
memberMarcos Meli31-Mar-11 6:42 
GeneralSome missing functionality Pin
bdaniel721-Mar-11 23:44
memberbdaniel721-Mar-11 23:44 
QuestionI have a CSV file where the Date and the time are into 2 fields. How can I use FileHelpers to aggregate the 2 fields into the same DateTime data ? Pin
Tintin9216-Mar-11 13:34
memberTintin9216-Mar-11 13:34 
QuestionHeaderText Pin
Wbehning11-Mar-11 9:04
memberWbehning11-Mar-11 9:04 
QuestionCompare To LINQToCSV Pin
Peter Lee1-Feb-11 7:37
memberPeter Lee1-Feb-11 7:37 
AnswerRe: Compare To LINQToCSV Pin
Peter Lee1-Feb-11 7:39
memberPeter Lee1-Feb-11 7:39 
GeneralMy vote of 5 Pin
Sunasara Imdadhusen30-Dec-10 20:16
memberSunasara Imdadhusen30-Dec-10 20:16 
GeneralMy vote of 5 Pin
sellinger3-Dec-10 0:38
membersellinger3-Dec-10 0:38 
GeneralError with 2.1 GB txt file Pin
chadok2-Dec-10 5:15
memberchadok2-Dec-10 5:15 
GeneralRe: Error with 2.1 GB txt file Pin
Marcos Meli2-Dec-10 6:02
memberMarcos Meli2-Dec-10 6:02 
GeneralRe: Error with 2.1 GB txt file Pin
chadok2-Dec-10 6:13
memberchadok2-Dec-10 6:13 
GeneralRe: Error with 2.1 GB txt file Pin
chadok2-Dec-10 6:42
memberchadok2-Dec-10 6:42 
GeneralRe: Error with 2.1 GB txt file Pin
Marcos Meli2-Dec-10 7:20
memberMarcos Meli2-Dec-10 7:20 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150819.1 | Last Updated 17 Aug 2015
Article Copyright 2005 by Marcos Meli
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid