Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
See more: C#
I have CSV File which size approx 200 to 500 mb and apply some validation then row insert and update into mysql database.
 
so I think, First CSV file into dataset and work on it.
 
is it good or other think is better than.
 
I need all work done in memory then i will pass data into database.
Posted 31-Jan-13 6:23am
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 1

I would use a streamreader
 
http://msdn.microsoft.com/en-us/library/system.io.streamreader.aspx[^]
 
Then validate the rows as i am reading the file, then you could you build your insert statements off of that
 
http://www.w3schools.com/sql/sql_insert.asp[^]
 
and insert them to your database,
  Permalink  
Comments
abhishek from jodhpur at 31-Jan-13 13:03pm
   
can i read csv in parts ,means like csv have 1 lake rows then i take data from csv in branch to 10k rows
abhishek from jodhpur at 31-Jan-13 13:08pm
   
and streamreader is not better in my case dataset and datatable is good
DinoRondelly at 31-Jan-13 13:10pm
   
Then use the streamreader and populate your datatable ... how are you going to populate the table if you dont read the file?
abhishek from jodhpur at 31-Jan-13 13:13pm
   
me use dataset place streamreader
DinoRondelly at 31-Jan-13 13:15pm
   
So either way you are using a stream reader,
abhishek from jodhpur at 31-Jan-13 13:14pm
   
i try to load csv in dataset then i work on that
but
now i think ,break the csv and me work on parts
DinoRondelly at 31-Jan-13 13:16pm
   
Going to have to read the file either way
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 2

First of all, if this is an occasional task, 0.5GB is not that much for the today's system (especially a server). But I suppose those 500MB are not coming from a single row. So you can simply parse it row by row and validate as needed. Or do you need some validation between the rows too?
Try using a ready-made CSV reader like this one: A Fast CSV Reader[^] or this one: http://blogs.msdn.com/b/jmstall/archive/2012/03/24/opensource-csv-reader-on-nuget.aspx[^]
  Permalink  
Comments
abhishek from jodhpur at 31-Jan-13 13:01pm
   
no i can't use ready-made csv
size goes then more then 500mb
Zoltán Zörgő at 31-Jan-13 13:04pm
   
And? None will fetch everything at once - and even if, you will probably have 500MB available... But they are using stream reading/line-by-line reading, thus size is no problem. But use TextReader if you have doubts.
What have you tried so far?
abhishek from jodhpur at 31-Jan-13 13:12pm
   
i try to load csv in dataset then i work on that
but
now i think ,break the csv and me work on parts
Zoltán Zörgő at 31-Jan-13 13:18pm
   
Than do it.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



Advertise | Privacy | Mobile
Web03 | 2.8.140827.1 | Last Updated 31 Jan 2013
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100