how to load the 1crore records from the csv file to the datatable in vb.net?

Question

0.00/5 (No votes)

See more:

Dear All,
Anyone tell me the efficient way to load the csv file which having more than 1 crore records. ie., the size of the file is more than 285MB. I've tried with oledb method, but i got Out of memory exception. Please help me to do this.

Regards,
Sathish

Posted 22-Nov-11 20:01pm

bluesathish

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Wayne Gaylard · Answer 1 · 2011-11-22T20:34:00

Solution 1

If I were you I would use a StreamReader, and read the file one line at a time, saving the data to a database as I went.

Hope this helps

Posted 22-Nov-11 20:34pm

Wayne Gaylard

Updated 22-Nov-11 21:17pm

v2

Comments

bluesathish 23-Nov-11 4:04am

No actuall i want to filter the input into two types, like datatable.select(column='value') this. Then the resultant datatables inserted to the database using sqlbulkcopy(). So i need to convert the input to the datatable. Please help me to do this.
I've tried StreamReader concept also the system getting hanging and went to not responding state. Please anyone help me to do this.

Regards,
Sathish

bluesathish 24-Nov-11 7:00am

Dear All,
I need to know the maximum size of the csv file in which Oledbconnection can process in VB.NET Application, I can tested with the csv file which has more than 12,00,000 records, its fine. but if i give the csv file which has 1crore records then i got the out of memory exception. Please anyone help me to do this.

Regards,
BlueSathish