Click here to Skip to main content
12,451,662 members (56,622 online)
Rate this:
 
Please Sign up or sign in to vote.
See more: C#
I have CSV File which size approx 200 to 500 mb and apply some validation then row insert and update into mysql database.

so I think, First CSV file into dataset and work on it.

is it good or other think is better than.

I need all work done in memory then i will pass data into database.
Posted 31-Jan-13 6:23am
Rate this: bad
 
good
Please Sign up or sign in to vote.

Solution 1

I would use a streamreader

http://msdn.microsoft.com/en-us/library/system.io.streamreader.aspx[^]

Then validate the rows as i am reading the file, then you could you build your insert statements off of that

http://www.w3schools.com/sql/sql_insert.asp[^]

and insert them to your database,
  Permalink  
Comments
abhishek from jodhpur 31-Jan-13 13:03pm
   
can i read csv in parts ,means like csv have 1 lake rows then i take data from csv in branch to 10k rows
abhishek from jodhpur 31-Jan-13 13:08pm
   
and streamreader is not better in my case dataset and datatable is good
DinoRondelly 31-Jan-13 13:10pm
   
Then use the streamreader and populate your datatable ... how are you going to populate the table if you dont read the file?
abhishek from jodhpur 31-Jan-13 13:13pm
   
me use dataset place streamreader
DinoRondelly 31-Jan-13 13:15pm
   
So either way you are using a stream reader,
abhishek from jodhpur 31-Jan-13 13:14pm
   
i try to load csv in dataset then i work on that
but
now i think ,break the csv and me work on parts
DinoRondelly 31-Jan-13 13:16pm
   
Going to have to read the file either way
Rate this: bad
 
good
Please Sign up or sign in to vote.

Solution 2

First of all, if this is an occasional task, 0.5GB is not that much for the today's system (especially a server). But I suppose those 500MB are not coming from a single row. So you can simply parse it row by row and validate as needed. Or do you need some validation between the rows too?
Try using a ready-made CSV reader like this one: A Fast CSV Reader[^] or this one: http://blogs.msdn.com/b/jmstall/archive/2012/03/24/opensource-csv-reader-on-nuget.aspx[^]
  Permalink  
Comments
abhishek from jodhpur 31-Jan-13 13:01pm
   
no i can't use ready-made csv
size goes then more then 500mb
Zoltán Zörgő 31-Jan-13 13:04pm
   
And? None will fetch everything at once - and even if, you will probably have 500MB available... But they are using stream reading/line-by-line reading, thus size is no problem. But use TextReader if you have doubts.
What have you tried so far?
abhishek from jodhpur 31-Jan-13 13:12pm
   
i try to load csv in dataset then i work on that
but
now i think ,break the csv and me work on parts
Zoltán Zörgő 31-Jan-13 13:18pm
   
Than do it.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Top Experts
Last 24hrsThis month


Advertise | Privacy | Mobile
Web02 | 2.8.160826.1 | Last Updated 31 Jan 2013
Copyright © CodeProject, 1999-2016
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100