Better option to quickly read all the XML files from folder and update the database (SQL SERVER 2014)

Question

1.00/5 (1 vote)

See more:

, +

I have few folders, each of which has millions of xml files. I need to read all the xml files from these folders and update database, which has to happen very quickly.

What I have tried:

Currently I am throwing multiple threads on the folder and processing the xml files.
This works well for less volume, but not for millions of xml files.
I have few high priority xml files in between these millions of xml files, which I do not know. These needs to be updated fast. Since I do not know which are the high priority files I am forced to update everything.

The area I am looking to improve is updating the database process. When all the threads are trying to update the database simultaneously, there is a performance issue.

Any better solution is appreciated.

Posted 26-Sep-17 3:14am

Member 11936418

Updated 26-Sep-17 20:28pm

v5

Add a Solution

Comments

Mehdi Gholam 26-Sep-17 10:36am

Which part of the current process is slow?

Richard MacCutchan 26-Sep-17 11:07am

Millions of files will take millions of time intervals to process. And running in multiple threads is not likely to have a significant impact on the processing time.

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Graeme_Grant · Accepted Answer · 2017-09-26T14:36:00

Your computer only has a limited number of physical & logical cores. Only a limited number of parallel processes can be executed. Throwing more threaded processes than cores at a problem can degrade performance not improve it. You need to throw more hardware (processing cores) to improve performance. Either more expensive CPU with more cores or more physical computers. Storage and how you use it can also slow you down.

If you don't want to invest in physical hardware, then you could use a cloud service provider like Azure - pay for only the cpu time that you use. Set the type of machine, the number of machines required, then let it run...