how workout with large data and apply some validations

Question

0.00/5 (No votes)

See more:

I have CSV File which size approx 200 to 500 mb and apply some validation then row insert and update into mysql database.

so I think, First CSV file into dataset and work on it.

is it good or other think is better than.

I need all work done in memory then i will pass data into database.

Posted 31-Jan-13 6:23am

abhishek from jodhpur

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

DinoRondelly · Answer 1 · 2013-01-31T06:27:00

Solution 1

I would use a streamreader

http://msdn.microsoft.com/en-us/library/system.io.streamreader.aspx[^]

Then validate the rows as i am reading the file, then you could you build your insert statements off of that

http://www.w3schools.com/sql/sql_insert.asp[^]

and insert them to your database,

Posted 31-Jan-13 6:27am

DinoRondelly

Comments

abhishek from jodhpur 31-Jan-13 13:03pm

can i read csv in parts ,means like csv have 1 lake rows then i take data from csv in branch to 10k rows

abhishek from jodhpur 31-Jan-13 13:08pm

and streamreader is not better in my case dataset and datatable is good

DinoRondelly 31-Jan-13 13:10pm

Then use the streamreader and populate your datatable ... how are you going to populate the table if you dont read the file?

abhishek from jodhpur 31-Jan-13 13:13pm

me use dataset place streamreader

DinoRondelly 31-Jan-13 13:15pm

So either way you are using a stream reader,

abhishek from jodhpur 31-Jan-13 13:14pm

i try to load csv in dataset then i work on that
but
now i think ,break the csv and me work on parts

DinoRondelly 31-Jan-13 13:16pm

Going to have to read the file either way

**Zoltán Zörgő** · Answer 2 · 2013-01-31T06:34:00

Solution 2

First of all, if this is an occasional task, 0.5GB is not that much for the today's system (especially a server). But I suppose those 500MB are not coming from a single row. So you can simply parse it row by row and validate as needed. Or do you need some validation between the rows too?
Try using a ready-made CSV reader like this one: A Fast CSV Reader[^] or this one: http://blogs.msdn.com/b/jmstall/archive/2012/03/24/opensource-csv-reader-on-nuget.aspx[^]

Posted 31-Jan-13 6:34am

Zoltán Zörgő

Comments

abhishek from jodhpur 31-Jan-13 13:01pm

no i can't use ready-made csv
size goes then more then 500mb

Zoltán Zörgő 31-Jan-13 13:04pm

And? None will fetch everything at once - and even if, you will probably have 500MB available... But they are using stream reading/line-by-line reading, thus size is no problem. But use TextReader if you have doubts.
What have you tried so far?

abhishek from jodhpur 31-Jan-13 13:12pm

i try to load csv in dataset then i work on that
but
now i think ,break the csv and me work on parts

Zoltán Zörgő 31-Jan-13 13:18pm

Than do it.