How to insert data from thousands of sources in one table

Question

1.00/5 (2 votes)

See more:

Hello Everyone

I have a requirement where I have to get info from almost 10000 sources over net and store it in a single table in fly time. These resources u can say are like millions of devices having unique id and I need to pump data into a single table in PostgreSQL.

What can be best possible approach to do this so that it doesn't degrades the performance of database and also doesn't acquire any lock as well to impact other transaction as well.

Posted 30-Jun-15 2:16am

Member 11802905

Updated 30-Jun-15 2:19am

v2

Add a Solution

Comments

Suvendu Shekhar Giri 30-Jun-15 8:32am

10000 sources? What kind of sources those are?

Member 11802905 30-Jun-15 8:40am

these are different media devices like TV or some other player from which I need to get media playing information as well as hardware related info.
So what can be best approach to insert these details data in a single table on fly time

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Mehdi Gholam · Answer 1 · 2015-06-30T04:29:00

Performance depends on many factors, but in your case you should specify the input rate and how you are going to query the data first.

Depending on those two factors your indexing strategy or the use of relational databases in general, will change.

In any case with the limited information you have provided, the only valid answer is try and see.