Click here to Skip to main content
14,579,188 members

Welcome to the Lounge

   

For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?

The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.
 
GeneralRe: Pakistan has an army of dinosaurs! Pin
ZurdoDev29-Jun-20 3:10
mveZurdoDev29-Jun-20 3:10 
JokeRe: Pakistan has an army of dinosaurs! Pin
Afzaal Ahmad Zeeshan29-Jun-20 5:13
mvaAfzaal Ahmad Zeeshan29-Jun-20 5:13 
GeneralRe: Pakistan has an army of dinosaurs! Pin
CodeWraith29-Jun-20 6:13
MemberCodeWraith29-Jun-20 6:13 
GeneralRe: Pakistan has an army of dinosaurs! Pin
RedDk29-Jun-20 7:06
MemberRedDk29-Jun-20 7:06 
GeneralRe: Pakistan has an army of dinosaurs! Pin
CodeWraith29-Jun-20 7:26
MemberCodeWraith29-Jun-20 7:26 
GeneralRe: Pakistan has an army of dinosaurs! Pin
RedDk29-Jun-20 8:32
MemberRedDk29-Jun-20 8:32 
GeneralRe: Pakistan has an army of dinosaurs! Pin
Kris321129-Jun-20 22:35
professionalKris321129-Jun-20 22:35 
GeneralAdventures in Async Pin
Jörgen Andersson29-Jun-20 0:02
professionalJörgen Andersson29-Jun-20 0:02 
Never bothered with Async programming before since I never needed it.
But now I'm having to take care of a weekly delivery of an 80 GB (eighty gigabyte) large XML-file.

The parsing and saving 10 million records to 30 different tables in a database takes more than an hour and there's no simple optimization left to do.
But I only use one kernel in the processor, so let's go parallell, it'll be fun learning. Right?

Easiest part is bulk copying to the database in parallel. Easy enough but it only shaves five minutes from the total time. This is not where the biggest bottleneck is.
The biggest bottleneck is the actual parsing of the XML.

I don't want to rework the whole application into using locks and thread-safe collections so I decide to split the work vertically instead. Add a task for every collection of data.
Also easy enough, now the processor is working close to 100%, but it takes twice as long. WTF | :WTF:

Apparently the creation of tasks has more overhead than the parsing of the data itself. Laugh | :laugh:

No shortcuts for me today. Back to the drawing board.
Wrong is evil and must be defeated. - Jeff Ello

GeneralRe: Adventures in Async PinPopular
OriginalGriff29-Jun-20 0:12
mveOriginalGriff29-Jun-20 0:12 
GeneralRe: Adventures in Async Pin
Jörgen Andersson29-Jun-20 0:26
professionalJörgen Andersson29-Jun-20 0:26 
GeneralRe: Adventures in Async Pin
OriginalGriff29-Jun-20 0:36
mveOriginalGriff29-Jun-20 0:36 
GeneralRe: Adventures in Async Pin
Kirk Hawley29-Jun-20 7:52
professionalKirk Hawley29-Jun-20 7:52 
GeneralRe: Adventures in Async Pin
Leo5629-Jun-20 21:31
MemberLeo5629-Jun-20 21:31 
GeneralRe: Adventures in Async Pin
dandy7229-Jun-20 5:24
Memberdandy7229-Jun-20 5:24 
PraiseRe: Adventures in Async Pin
Member 1467846130-Jun-20 6:41
MemberMember 1467846130-Jun-20 6:41 
GeneralRe: Adventures in Async Pin
Kornfeld Eliyahu Peter29-Jun-20 0:13
mveKornfeld Eliyahu Peter29-Jun-20 0:13 
GeneralRe: Adventures in Async Pin
Jörgen Andersson29-Jun-20 0:15
professionalJörgen Andersson29-Jun-20 0:15 
GeneralRe: Adventures in Async Pin
Kornfeld Eliyahu Peter29-Jun-20 0:35
mveKornfeld Eliyahu Peter29-Jun-20 0:35 
GeneralRe: Adventures in Async Pin
Jörgen Andersson29-Jun-20 0:52
professionalJörgen Andersson29-Jun-20 0:52 
GeneralRe: Adventures in Async Pin
Richard Deeming29-Jun-20 0:30
mveRichard Deeming29-Jun-20 0:30 
GeneralRe: Adventures in Async Pin
Jörgen Andersson29-Jun-20 0:38
professionalJörgen Andersson29-Jun-20 0:38 
GeneralRe: Adventures in Async Pin
#realJSOP29-Jun-20 2:18
mva#realJSOP29-Jun-20 2:18 
GeneralRe: Adventures in Async Pin
Greg Utas29-Jun-20 0:42
professionalGreg Utas29-Jun-20 0:42 
GeneralRe: Adventures in Async Pin
Jörgen Andersson29-Jun-20 0:46
professionalJörgen Andersson29-Jun-20 0:46 
GeneralRe: Adventures in Async Pin
Garth J Lancaster29-Jun-20 1:04
professionalGarth J Lancaster29-Jun-20 1:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.