Quering very large .out (flat file)

Question

0.00/5 (No votes)

See more:

Hi,
Could someone help me on following issue?
I have to read a particular field(s) from a flat file which has millions of records. Extension of flat file is .out.
Which is the best way to query such files on performance wise?
As of now we are thinking LINQ or XSLT parsing.
Please suggest your ideas on above issue.
Thanks
Prasad

Posted 4-Aug-10 2:39am

2000 P

Add a Solution

2 solutions

Solution 1

You could use LINQ but you would first have to read the lines from the file and parse them into a list before you could use it. Another option is to use the bulk import function of SQLServer and import the data into a table that way.

http://msdn.microsoft.com/en-us/library/ms187042.aspx[^]

http://msdn.microsoft.com/en-us/library/ms188365.aspx[^]

Reading it into memory could be a problem because of the number of records and memory limitations. This does of course depend on the size of the lines and the data you actually need because the rest can be discarded. You could also choose to only hold in memory the columns you need in your where clause and later on read the data you want to select based on the record/line numbers the select returned.

Good luck!

Posted 4-Aug-10 3:59am

E.F. Nijboer

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Andrew Rissing · Accepted Answer · 2010-08-04T04:17:00

Depending on the format of the flat file, I would recommend one of the following:

1) A Fast CSV Reader[^] - Handles delimited file formats

2) A Portable and Efficient Generic Parser for Flat Files[^] - Handles delimited and fixed width file formats

Both will allow you to get the data into memory in a quick way. Depending on what you're doing with it though, you may prefer one of the methods Nijboer has already suggested.