Click here to Skip to main content
15,889,815 members
Home / Discussions / C#
   

C#

 
GeneralRe: Parse 17,000 text files and insert into database Pin
OriginalGriff15-Nov-16 8:17
mveOriginalGriff15-Nov-16 8:17 
GeneralRe: Parse 17,000 text files and insert into database Pin
Gerry Schmitz15-Nov-16 8:19
mveGerry Schmitz15-Nov-16 8:19 
GeneralRe: Parse 17,000 text files and insert into database Pin
peterkmx15-Nov-16 8:50
professionalpeterkmx15-Nov-16 8:50 
AnswerRe: Parse 17,000 text files and insert into database Pin
Philippe Mori15-Nov-16 8:11
Philippe Mori15-Nov-16 8:11 
AnswerRe: Parse 17,000 text files and insert into database Pin
Nathan Minier15-Nov-16 8:33
professionalNathan Minier15-Nov-16 8:33 
AnswerRe: Parse 17,000 text files and insert into database Pin
Mycroft Holmes15-Nov-16 11:54
professionalMycroft Holmes15-Nov-16 11:54 
AnswerRe: Parse 17,000 text files and insert into database Pin
Bernhard Hiller15-Nov-16 21:33
Bernhard Hiller15-Nov-16 21:33 
AnswerRe: Parse 17,000 text files and insert into database Pin
Jimmanuel23-Nov-16 10:48
Jimmanuel23-Nov-16 10:48 
I recently had a similar problem and gained some speed with a 2 step change: parallelize the
C#
foreach (string f in files)
and then ditch the file streaming in favor of File.ReadAllLines(...). Basically the disk was my bottleneck and it performed better doing a bulk reads of data instead of streaming line by line; after each bulk read I had a chunk of data in memory that I could process bound only by the limitations of my CPU. This does, of course, assume that you have some memory to spare because you're reading entire files in at a time. There was also a limit though as to how many concurrent file reads I could throw at my disk, after a certain amount the performance dropped significantly so I had to throttle back how many parallel file reads my app could do. I did some trial and error to find the best number, YMMV.

That would also require you to redo your database access though since you probably don't want to be doing that from multiple threads, but that kind of goes along with some of the other suggestions you've received. Collecting that data in memory and do a bulk insert later on instead of individual inserts would make a big difference, too.
Badger | [badger,badger,badger,badger...]

QuestionDevelop Web Service REST Pin
Member 1283622014-Nov-16 3:37
Member 1283622014-Nov-16 3:37 
QuestionTextbox updation Pin
Avinash.Banda14-Nov-16 0:24
Avinash.Banda14-Nov-16 0:24 
AnswerRe: Textbox updation Pin
OriginalGriff14-Nov-16 0:31
mveOriginalGriff14-Nov-16 0:31 
GeneralRe: Textbox updation Pin
Avinash.Banda14-Nov-16 0:36
Avinash.Banda14-Nov-16 0:36 
GeneralRe: Textbox updation Pin
OriginalGriff14-Nov-16 0:39
mveOriginalGriff14-Nov-16 0:39 
GeneralRe: Textbox updation Pin
Avinash.Banda14-Nov-16 0:43
Avinash.Banda14-Nov-16 0:43 
GeneralRe: Textbox updation Pin
OriginalGriff14-Nov-16 0:45
mveOriginalGriff14-Nov-16 0:45 
GeneralRe: Textbox updation Pin
Avinash.Banda14-Nov-16 0:52
Avinash.Banda14-Nov-16 0:52 
GeneralRe: Textbox updation Pin
OriginalGriff14-Nov-16 1:03
mveOriginalGriff14-Nov-16 1:03 
GeneralRe: Textbox updation Pin
Avinash.Banda14-Nov-16 1:04
Avinash.Banda14-Nov-16 1:04 
QuestionCan I send Push Notifications to mobile phones from Windows Forms Application? Pin
Onur ERYILMAZ13-Nov-16 23:58
Onur ERYILMAZ13-Nov-16 23:58 
AnswerRe: Can I send Push Notifications to mobile phones from Windows Forms Application? Pin
Eddy Vluggen14-Nov-16 3:51
professionalEddy Vluggen14-Nov-16 3:51 
QuestionFile Transfer Queue Architecture Pin
Kevin Marois10-Nov-16 6:16
professionalKevin Marois10-Nov-16 6:16 
QuestionRe: File Transfer Queue Architecture Pin
Gerry Schmitz10-Nov-16 6:41
mveGerry Schmitz10-Nov-16 6:41 
AnswerRe: File Transfer Queue Architecture Pin
Kevin Marois10-Nov-16 6:42
professionalKevin Marois10-Nov-16 6:42 
GeneralRe: File Transfer Queue Architecture Pin
Gerry Schmitz10-Nov-16 7:04
mveGerry Schmitz10-Nov-16 7:04 
GeneralRe: File Transfer Queue Architecture Pin
Kevin Marois10-Nov-16 7:22
professionalKevin Marois10-Nov-16 7:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.