Click here to Skip to main content
15,434,217 members
Articles / Programming Languages / C#
Article
Posted 1 Jul 2006

Stats

1.5M views
13.8K downloads
211 bookmarked

Fast Excel file reader with basic functionality

Rate me:
Please Sign up or sign in to vote.
4.88/5 (104 votes)
1 Jul 20063 min read
A set of managed classes for native reading of data from MS Excel files

Introduction

When I was writing one of the projects (multi-format data parser) I had to deal with Microsoft Excel files.

First of all I tried to read data via COM interop, but this way was very slow (and sometimes input files have up to 100,000 rows and even more) and required Excel to be installed on the target machine.

Then I tried to read data via Excel's OleDb driver. When DataReader ran through data rows without reading them (i.e. while (rd.Read());) it was quite fast, but retrieving data was still very slow.

When I had tried out one of commercial products (TMS FlexCel Studio), it showed rather good results (several seconds for loading 17 MB file, 2 sheets, ~ 80 000 rows), but access to them was not very fast (at least not suitable for displaying data using DataGridView in virtual mode).

So, I decided to write my own reader for Excel files.

Background

First of all I decided that the reader will produce a DataSet object, with sheets represented as DataTables within it. Access to DataTable is rather fast, and its TableName property is suitable for sheet title.

To read an Excel file, we need to get BIFF stream from xls OLE2 container. Working with OLE2 container is implemented in the following classes:

  • XlsHeader (file header representation)
  • XlsFat (support for file system in OLE2 container)
  • XlsRootDirectory (directory tree representation)
  • XlsDirectoryEntry, and
  • XlsStream (file stream)

Note that implementation of OLE2 container object model is limited (e.g. I didn't write any MiniFAT support code) and Excel-specific (XlsHeader supports only MS Excel header field values, any other will produce InvalidHeaderException).

Now we can parse BIFF stream with spreadsheet data. For handy reading data from stream we have a XlsBiffStream class derived from XlsStream. It provides a set of methods for reading XlsBiffRecord common structures. And then, each structure is converted to the appropriate type. Note that only a few record types are supported. All other structures are read as basic XlsBiffRecord and are ignored.

My implementation of reader ignores any text formatting, graphics, embedded objects and charts. Only cell values are read. When a cell contains a formula, the last calculated formula value is used as the cell value. Also, only indexed sheets are read (i.e. sheets with index data) - I didn't write not-indexed sheets support due to my laziness (I didn't come across any file without an index yet).

Using the code

Now, it's very easy to use the parser. All you need is a Stream with an Excel file.

C#
FileStream fs = new FileStream(@"c:\file.xls", 
        FileMode.Open, FileAccess.Read);
ExcelDataReader rd = new ExcelDataReader(fs);
fs.Close();

// Now we can access data:
DataSet data = rd.WorkbookData;

Points Of Interest

Latest versions of Excel use SST (Shared String Table) to store string information, and text cells only reference string by index. Also, the size of one BIFF section is limited, and SST can be rather big, so it uses the so-called CONTINUE sections. Every string in SST can be saved as Unicode (2 byte) or as ANSI (single byte per symbol). But when the string is broken with the continue section, it can change its encoding from one to another! This took me a lot of debugging, and I wish nobody has to do the same again...

Results

This implementation of reader is rather nonoptimal (I was pressed for time), but it showed a good speed. It surely beats any interop or OLE method and even some commercial analogs.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
Web Developer
Russian Federation Russian Federation
Alex
.NET Developer
Russian Federation
rakemaker@gmail.com

Comments and Discussions

 
QuestionRe: Full Source Code Pin
Deepankar10-Aug-06 7:00
MemberDeepankar10-Aug-06 7:00 
QuestionHow to read open Excel files? [modified] Pin
93Current24-Jul-06 13:47
Member93Current24-Jul-06 13:47 
AnswerRe: How to read open Excel files? Pin
93Current25-Jul-06 0:19
Member93Current25-Jul-06 0:19 
GeneralRequirements Pin
ram_gk10-Jul-06 16:41
Memberram_gk10-Jul-06 16:41 
GeneralRe: Requirements Pin
Messir11-Jul-06 3:13
MemberMessir11-Jul-06 3:13 
GeneralFound one bug [modified] Pin
Messir9-Jul-06 9:26
MemberMessir9-Jul-06 9:26 
GeneralGreat job Pin
egozi138-Jul-06 10:50
Memberegozi138-Jul-06 10:50 
GeneralNice piece of code but had to make some changes Pin
dsmile7-Jul-06 9:22
Memberdsmile7-Jul-06 9:22 
Was looking for something like this, started to use it and it works quite well.

There are some adjustments I made because there were problems with the formula fields when I was using it. I have to use the calculated values from it for an import.

- I changed the encoding for formulafields from unicode to default/ascii in ExcelDataReader.ReadWorkSheet

case BIFFRECORDTYPE.FORMULA:<br />
        case BIFFRECORDTYPE.FORMULA_OLD:<br />
        ((XlsBiffFormulaCell)cell).UseEncoding = m_encoding;


into:

case BIFFRECORDTYPE.FORMULA:<br />
        case BIFFRECORDTYPE.FORMULA_OLD:<br />
        ((XlsBiffFormulaCell)cell).UseEncoding = System.Text.Encoding.Default;


and i changed in ExcelDataReader.Core.XlsBiffFormulaString.Value

public string Value<br />
  {<br />
       get { return m_UseEncoding.GetString(m_bytes, m_readoffset + ((m_UseEncoding.IsSingleByte) ? 2 : 3), Length * ((m_UseEncoding.IsSingleByte) ? 1 : 2)); }<br />
  }


into:

public string Value<br />
  {<br />
       get { return m_UseEncoding.GetString(m_bytes, m_readoffset + 2 + ((m_UseEncoding.IsSingleByte) ? 1 : 2), Length * ((m_UseEncoding.IsSingleByte) ? 1 : 2)); }<br />
  }


The normal formula fields seem to work for me now, but found out shared formula fields still need implementation too for me to be ok. (BIFF record type SHRFMLA) So guess I'll dig up some documentation on that and give a go on implementing it.




Ferdy Bosma
Bosma IT
The Netherlands
GeneralRe: Nice piece of code but had to make some changes Pin
pleury12-Nov-06 17:44
Memberpleury12-Nov-06 17:44 
GeneralRe: Nice piece of code but had to make some changes Pin
ecunha19-Sep-14 5:35
Memberecunha19-Sep-14 5:35 
GeneralThanks for the work Pin
Brian Leach5-Jul-06 8:36
MemberBrian Leach5-Jul-06 8:36 
GeneralXML or excel spread sheet? [modified] Pin
Keith Vinson5-Jul-06 5:35
MemberKeith Vinson5-Jul-06 5:35 
GeneralRe: XML or excel spread sheet? Pin
Messir5-Jul-06 5:56
MemberMessir5-Jul-06 5:56 
GeneralRe: XML or excel spread sheet? Pin
Keith Vinson5-Jul-06 6:09
MemberKeith Vinson5-Jul-06 6:09 
GeneralExcel Data Writer Pin
BradleyC4-Jul-06 4:44
MemberBradleyC4-Jul-06 4:44 
GeneralRe: Excel Data Writer Pin
Messir4-Jul-06 6:25
MemberMessir4-Jul-06 6:25 
GeneralHyperlink Pin
kuerbis2-Jul-06 10:41
Memberkuerbis2-Jul-06 10:41 
GeneralRe: Hyperlink Pin
Messir2-Jul-06 18:14
MemberMessir2-Jul-06 18:14 
GeneralNice Article Pin
Paul Conrad1-Jul-06 17:58
professionalPaul Conrad1-Jul-06 17:58 
GeneralRe: Nice Article Pin
Messir1-Jul-06 18:48
MemberMessir1-Jul-06 18:48 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.