How do I optimize my VB.NET csv import

Question

0.00/5 (No votes)

See more:

Hi Guys

I would like to know what would be the best way to handle imports of csv files, with large rows of data(between 200 000 - 1 200 000 records).

Initially when i built the module the files went up to around 20 000 records.

I am currently using textfield passer and then while reading each record add the record to the object and add the object to the database.

Issue is when there is larger imports like now imports are taking over 15 - 20 min to import. which i now need to optimize

I am using an oralce database

What I have tried:

What i have now tried is reading the records into a datatable and then trying a bulk insert of the datatable, Just would like to know if there is a better way to handle the bulk imports

Posted 2-Oct-16 21:11pm

IsiR_19

Updated 3-Oct-16 0:19am

Add a Solution

Comments

Mehdi Gholam 3-Oct-16 3:42am

Show your code.

Sinisa Hajnal 3-Oct-16 4:26am

You could try to split the file into several related files and run import threads in parallel?

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

#realJSOP · Answer 1 · 2016-10-03T00:19:00

Solution 1

Look at my recent article. It should be quite simple to convert to VB.Net.

CSV File Parser[^]

You could also just put it into a DLL assembly and call it from your VB app.

As far as bulk inserts go, I would write a stored procedure and submit the records one at a time. That would be faster.

Posted 3-Oct-16 0:19am

#realJSOP

Updated 3-Oct-16 0:20am

v2