Validating data with Flat File Checker

• Download source code - 303 KB
• Download console version code as code example - 104 KB
• Download Flat File Checker application installer [SourceForge]

Introduction

It is often required to validate flat files before importing them to the database, or after exporting data from the database. Flat File Checker allows you to validate an inter-related set of flat files (*.csv, *.psv, *.txt, etc.) without having to parse them directly in your code.

The business rules against which you must validate the files must be made available to the FLaFi library in the form of an XML file, which is created through the graphical user interface of the FlaFi application. Here are examples of such rules:

• Value in the field is Required
• Field is a date with a given format and within a given range
• Field structure must match a Regular Expression
• Two files are linked [relational links]
• Field must be compared to another field
• Unique constraint is imposed on one or several fields
• Validation must be made against values stored in a database

The task of validation can be now addressed in a few easy steps:

• Create a Flat File Schema (XML file that contains business/data rules for files that you want to validate)
• Add the Flat File Checker library to your .NET application
• Run validation with minimum code within your application

Background

The idea behind Flat File Checker is to create an application that can validate structured data in a text file based on schema stored in XML, in a way similar to the way XML files are validated. This approach separates data/business rules from the validation itself, and thus considerably less coding is required.

Creating the schema

Here is a quick way to create a schema file to use for validation by FlaFi:

• Download the installer of the latest version of Flat File Checker from SourcForge.net
• Install the application and start the application
• Create a new schema using the application and save it locally in an XML file

When the schema is created, you can view it in browser as a readable data exchange specification:

Add the Flat File Library to your solution

Add the Flat File Library (see link at the top) project to your solution in Visual Studio.

Run the validation and process errors

To use the Flat File Library, you will need to use these main classes:

• FlatFileSchema - The class that contains a collection of files
• File - The virtual class that gives access to several objects associated with a flat file: columns, errors, etc.
• DataError - The class that contains details of the data error

You can also include system threading to your project as validation is done in asynchronous mode:

Imports System.Threading

The following code shows how to start validation (the RunValidation sub) and process errors if needed (the FileSetValidated delegate sub).

'-----------------------------------------------------------
'                 VB.Net Example
'-----------------------------------------------------------
Private WithEvents _files As FlatFileSchema
Private do_checks As AutoResetEvent
' Start validation
Sub RunValidation()
 ' Use Flat File Checker user interface to create Schema file.
    _files = New FlatFileSchema(<Schema file path>)
      do_checks=_files.RunChecks()
End Sub

' Feedback errors when file(s) validation is complete
Private Sub FileSetValidated(ByVal sender As Object, _
ByVal e As SchemaValidatedEventArgs) _
                  Handles _files.Validated
 Dim file As FlatFile
 Dim err As DataError
 For Each file In _files.Files
       For Each err In file.Errors
          ' Feed back error to users here
          ...
       Next err
 Next file

End Sub

Points of interest

The console version of Flat File Checker (code attached) makes it possible to run validation from command line with just one line:

c:/program files/flat file checker>flafi.exe <schema path>