Click here to Skip to main content
15,896,557 members

Welcome to the Lounge

   

For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?

The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.

 
GeneralRe: Keurig (new tech), you got it wrong! Pin
raddevus18-Feb-19 9:59
mvaraddevus18-Feb-19 9:59 
GeneralRe: Keurig (new tech), you got it wrong! Pin
theoldfool18-Feb-19 11:22
professionaltheoldfool18-Feb-19 11:22 
GeneralRe: Keurig (new tech), you got it wrong! Pin
raddevus19-Feb-19 1:42
mvaraddevus19-Feb-19 1:42 
GeneralRe: Keurig (new tech), you got it wrong! Pin
OriginalGriff18-Feb-19 11:22
mveOriginalGriff18-Feb-19 11:22 
GeneralRe: Keurig (new tech), you got it wrong! Pin
raddevus19-Feb-19 1:43
mvaraddevus19-Feb-19 1:43 
GeneralRe: Keurig (new tech), you got it wrong! Pin
j snooze18-Feb-19 11:46
j snooze18-Feb-19 11:46 
GeneralRe: Keurig (new tech), you got it wrong! Pin
raddevus19-Feb-19 1:44
mvaraddevus19-Feb-19 1:44 
GeneralDid you know, final PinPopular
#realJSOP18-Feb-19 8:15
professional#realJSOP18-Feb-19 8:15 
I started with multi-gb size XML files that would cause out-of-memory exceptions.

After I resolved that issue (by using XmlTextReader/XElement instead of XDocument), it was taking over 20 minutes to process the sample file set (434,000 rows). This is where I was on Friday.

I added some TPL love and got it down to about 12 minutes.

I reworked the target tables. At first, I was creating a single monolithic flat table, and a lot of data was being repeated for every record. I created two tables (one that holds the repeating data, and one that holds the unique data), and got it down to just a little over 6 minutes.

This code will replace a Perl script that:

0) Creates a bunch of smaller XML files, probably so it doesn't run out of memory
1) Took over an hour to process the same set of sample files.
2) Did not return all of the data (because the stakeholder doesn't really need it)

Added benefits:

0) My code can process ZIP (extracts NESSUS files from them for processing) and bare naked NESSUS files
1) Processes multiple files with a single execution
2) Puts ALL of the data into the database

Caveat: Because of the limited hardware the app will actually be running on, processing time will take longer (because of the TPL's default behavior), but not significantly so, and CERTAINLY not take an hour.

It's been a good weekend. Smile | :)
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013

GeneralRe: Did you know, final Pin
OriginalGriff18-Feb-19 8:17
mveOriginalGriff18-Feb-19 8:17 
GeneralRe: Did you know, final Pin
#realJSOP18-Feb-19 9:29
professional#realJSOP18-Feb-19 9:29 
GeneralRe: Did you know, final Pin
Nelek18-Feb-19 23:59
protectorNelek18-Feb-19 23:59 
GeneralRe: Did you know, final Pin
#realJSOP19-Feb-19 1:54
professional#realJSOP19-Feb-19 1:54 
GeneralRe: Did you know, final Pin
dandy7219-Feb-19 8:42
dandy7219-Feb-19 8:42 
GeneralRe: Did you know, final Pin
RickZeeland18-Feb-19 9:31
mveRickZeeland18-Feb-19 9:31 
GeneralRe: Did you know, final Pin
Mark_Wallace18-Feb-19 9:34
Mark_Wallace18-Feb-19 9:34 
GeneralRe: Did you know, final Pin
#realJSOP18-Feb-19 10:06
professional#realJSOP18-Feb-19 10:06 
GeneralRe: Did you know, final Pin
Mark_Wallace18-Feb-19 10:31
Mark_Wallace18-Feb-19 10:31 
GeneralRe: Did you know, final Pin
pkfox18-Feb-19 10:50
professionalpkfox18-Feb-19 10:50 
GeneralRe: Did you know, final Pin
#realJSOP18-Feb-19 11:11
professional#realJSOP18-Feb-19 11:11 
GeneralRe: Did you know, final Pin
Jim Meadors18-Feb-19 19:37
Jim Meadors18-Feb-19 19:37 
GeneralRe: Did you know, final Pin
Gary Wheeler19-Feb-19 7:07
Gary Wheeler19-Feb-19 7:07 
GeneralRe: Did you know, final Pin
#realJSOP19-Feb-19 7:38
professional#realJSOP19-Feb-19 7:38 
GeneralRe: Did you know, final Pin
dandy7219-Feb-19 8:50
dandy7219-Feb-19 8:50 
GeneralThought of the Day Pin
OriginalGriff18-Feb-19 4:51
mveOriginalGriff18-Feb-19 4:51 
GeneralRe: Thought of the Day Pin
megaadam18-Feb-19 5:06
professionalmegaadam18-Feb-19 5:06 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.