Click here to Skip to main content
14,360,121 members
Rate this:
Please Sign up or sign in to vote.
See more:
Hi,
Could someone help me on following issue?
I have to read a particular field(s) from a flat file which has millions of records. Extension of flat file is .out.
Which is the best way to query such files on performance wise?
As of now we are thinking LINQ or XSLT parsing.
Please suggest your ideas on above issue.
Thanks
Prasad
Posted
Rate this:
Please Sign up or sign in to vote.

Solution 2

Depending on the format of the flat file, I would recommend one of the following:

1) A Fast CSV Reader[^] - Handles delimited file formats

2) A Portable and Efficient Generic Parser for Flat Files[^] - Handles delimited and fixed width file formats

Both will allow you to get the data into memory in a quick way. Depending on what you're doing with it though, you may prefer one of the methods Nijboer has already suggested.
   
Rate this:
Please Sign up or sign in to vote.

Solution 1

You could use LINQ but you would first have to read the lines from the file and parse them into a list before you could use it. Another option is to use the bulk import function of SQLServer and import the data into a table that way.

http://msdn.microsoft.com/en-us/library/ms187042.aspx[^]

http://msdn.microsoft.com/en-us/library/ms188365.aspx[^]

Reading it into memory could be a problem because of the number of records and memory limitations. This does of course depend on the size of the lines and the data you actually need because the rest can be discarded. You could also choose to only hold in memory the columns you need in your where clause and later on read the data you want to select based on the record/line numbers the select returned.

Good luck!
   

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)




CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100