LINQ to Text or CSV Files - Part 1

• Download source - 13.54 KB

Introduction

With the introduction of LINQ in C# 3.0, the life of a developer has become very easy when it comes to querying collections. In this article, I will show you a quick and easy way to query large CSV files.

Background

Imagine you have a very large CSV file. Loading the entire file into memory (e.g. DataSet) and querying it with LINQ would be a huge overhead. So I thought why not use StreamReader on the file instead.

Using the Code

I have implemented IEnumerable<string[]> and for every call to GetEnumerator, a line is read from CSV file. This class returns an enumerable string[].

public class TextFileReader : IEnumerable<string[]>
{
    private string _fileName = string.Empty;

    public TextFileReader(string fileName)
    {
        this._fileName = fileName;
    }

    #region IEnumerable<string[]> Members
  
    IEnumerator<string[]> IEnumerable<string[]>.GetEnumerator()
    {
        using (StreamReader streamReader = new StreamReader(this._fileName))
        {
            while (!streamReader.EndOfStream)
            {
                yield return streamReader.ReadLine().Split(new char[] { ',' });
            }
        }
    }

    #endregion
  
    #region IEnumerable Members
 
    IEnumerator IEnumerable.GetEnumerator()
    {
        IEnumerable<string[]> iEnumerable = this; 
        yield return iEnumerable.GetEnumerator();
    }
 
    #endregion
}

Using this class is very simple. Just initialize the class with the CSV file name and you are all set to LINQ it using standard operator. The following code shows how to query all the rows in the CSV file with the first column text starting with 'a'.

TextFileReader reader = new TextFileReader(@"Sample.txt");

var query = from it in reader
            where it[0].StartsWith("a")
            select new { Name = it[0], EmailAddress = it[1] };

foreach (var x in query)
{
    Console.WriteLine(String.Format("Name={0} EmailAddress = {1}", 
    x.Name, x.EmailAddress));
}

Points of Interest

Well, this may not be the best way to implement this. I am also into the process of learning LINQ. Please feel free to give your comments or suggestions. In my next article, I will show you how to write a custom query provider for CSV files. Please stay tuned.

History

• 11^th October, 2008: Initial post
• 14^th Novermber, 2008: Updates source and article