Database

Eddy Vluggen12-Feb-14 6:39

12-Feb-14 6:39

Agent_Spock wrote:
how much pain it will cause it to the developers doesn't matter

Now there's an helpfull attitude; aren't they the ones who have to support your sh*t?

Oracle, MySql and Sql Server can handle large amounts of data. Program against the IDb* interfaces and speak SQL92 - download the trials and TEST it. Have some console-app generate 100 million records if that's the kind of data you expect.

Don't expect to be databinding them to a gridview, regardless of your database. As soon as you have the test-setup, you'll also experience it's speed. Next you'll be learning what an index is, and find that all three db-servers support them. Aw, and yes, the schema does matter.

..and do take in account that the budget is a constraint; if you can afford a harddisk and Sql Server, you could also afford MySQL and an SSD-drive. Hard to predict who'd be faster out of those two Smile | :)

Agent_Spock wrote:
manipulate 100 million records which configures with c# win. form.

Did you say "manipulate"? Didn't you mean "store"?

--edit;
http://en.wikipedia.org/wiki/Comparison_of_relational_database_management_systems[^]
Hint: you'll need support for indexes, partitioning, and might want to take a look at both capabilities (do you need support for each type? each join?) and max. sizes of tables and database.

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

modified 12-Feb-14 13:04pm.

agent_kruger13-Feb-14 1:03

13-Feb-14 1:03

yes, i mean store and search. As my work is little tricky it has to search through 100 million records every hour.

modified 14-Feb-14 6:46am.

Jörgen Andersson13-Feb-14 2:09

Jörgen Andersson

13-Feb-14 2:09

You have to manipulate ALL of those 100 million records every hour?
Or is it just some, and then what percentage?
And what kind of reporting do you need to do? On how many records?

Fact is that it's not so important which DB you're having as long as you're just doing CRUD.
The size of the database is also not very important for the performance (Were talking about a factor of a couple of hundred times larger for every new level in an index, assuming B-Tree index).
It's the number of transactions! And how many records that are affected by the queries.

So the limiting factors will be the hardware (mostly the harddrives), the configuration of the hardware (It doesn't matter if the drives are fast if they're in a RAID5), and the indexes.
If you have no indexes at all, all inserts will be ridiculously fast, but querying or updating will be just as ridiculously slow.
Adding every index you can think about will make both inserts and updates ridiculously slow instead, while the queries will be faster.
Note that updates needs to both read and write.

Finding the right indexes will give you a good balance.

Wrong is evil and must be defeated.
- Jeff Ello[^]

Eddy Vluggen13-Feb-14 2:59

13-Feb-14 2:59

"Manipulate" 100 million records per hour?

Ask Amazon or Google to host your data.

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

agent_kruger13-Feb-14 4:38

13-Feb-14 4:38

sir, in this context i used "manipulate" for "searching"

Eddy Vluggen13-Feb-14 10:28

13-Feb-14 10:28

Then you are using the wrong terms. "Manipulate" means updating, inserting or deleting records. Reading doesn't manipulate the data, it fetches it. Selecting a few million should be doable, depending on the hardware and software-combination and the skillset of the dba.

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

agent_kruger14-Feb-14 0:47

14-Feb-14 0:47

sorry, for all this.i have corrected my reply.

Eddy Vluggen14-Feb-14 6:54

14-Feb-14 6:54

Databases are optimized to work with records; there will be a HUGE difference between searching records (select using a where) and searching a specific substring within a NTEXT field.

If you are going to search within the contents of the fiels, you'd be wanting a full-text search catalog. Again, supported by most major databases, but their speed may vary wildly.

Can you post a schema of the data that you'll be storing? Should I be thinking about simple data like measurements or prices? (lots o' fields with numbers) Or more toward text? (lots of short readable text-fields, like profiles), more toward memo's (single large textfield), or even documents (Word, PDF?)

In the case of documents I'd suggest to dump the files in the filesystem - and to use something like Google Desktop Search to search for specific terms.

Agent_Spock wrote:
sorry, for all this

Don't be; for someone who doesn't code all those things might sound roughly the same. Sorry for my short and blunt answers.

Live long and prosper Smile | :)

agent_kruger14-Feb-14 16:51

14-Feb-14 16:51

no sir, it consists of data which is converted into bytes.

Eddy Vluggen14-Feb-14 22:58

14-Feb-14 22:58

Everything that a computer stores is encoded in bytes; images, text, applications - they're all stored as bytes. Hence, the remark that it's going to store bytes is not very helpfull. That way I'd assume a large binary blob, and "searching" to be a series of bytes.

Those bytes represent something; data, in whatever form. What "kind" of data you're going to store determines the best approach.

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

agent_kruger15-Feb-14 1:36

15-Feb-14 1:36

no i am using it in such way that each record has max. 25 letters

Eddy Vluggen15-Feb-14 23:15