|
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)?
I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance.
Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database?
I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking.
Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it!
wreckless
|
|
|
|
|
I think you that be looking for a silver bullet number when the decision is based on a range of factors. I would place platform as the first criteria and would consider xml for mobile.
Complexity of the data structure would be the next criteria, 3-5 small tables and I might use XML (unlikely as I am very database centric) much more than that and I would not even consider xml.
As for volume of data, it would probably not even come into the equation.
This discussion should also probably not be in the Lounge, expect some left field answers!
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
There are no database rockstars, but those who master databases are a SELECT group.
|
|
|
|
|
ORDER, please!
No object is so beautiful that, under certain conditions, it will not look ugly. - Oscar Wilde
|
|
|
|
|
Duncan Edwards Jones wrote: There are no database rockstars,
Well, there is "the One". Oracle told me so.
Ok, I am out.
You have just been Sharapova'd.
|
|
|
|
|
Yeah, it's pretty hard to JOIN their RANKS! 
|
|
|
|
|
Indeed, a GROUP HAVING few members.
I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
|
|
|
|
|
It looks as I am BETWEEN monsters, WHERE was my head when I decided to JOIN? I'm about to DROP the USE of this GROUP BY heaven's sake!
BTW I dislike SQL and DBs
Geek code v 3.12
GCS d--- s-/++ a- C++++ U+++ P- L- E-- W++ N++ o+ K- w+++ O? M-- V? PS+ PE- Y+ PGP t++ 5? X R++ tv-- b+ DI+++ D++ G e++>+++ h--- r++>+++ y+++*
Weapons extension: ma- k++ F+2 X
I use 1TBS
|
|
|
|
|
Try using the KEY (the whole key, and nothing but the key)
|
|
|
|
|
I'm feeling left outer this conversation
PooperPig - Coming Soon
|
|
|
|
|
He could try, but he is not the right CANDIDATE to have the KEY.
You have just been Sharapova'd.
|
|
|
|
|
|
If you're not clever enough to join the RANKs[^], you could try the DENSE_RANKs[^]. 
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Yup, and we hang out in HEAPs and CLUSTERS.
If it's not broken, fix it until it is
|
|
|
|
|
here we go.... 
Charlie Gilley
<italic>Stuck in a dysfunctional matrix from which I must escape...
"Where liberty dwells, there is my country." B. Franklin, 1783
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
|
|
|
|
|
Databases do lots more than just store data. It's a centralized point for transactional data storage, modification and retrieval.
Try using some XML files in an enterprise environment. That's simply not going to work.
If you're just building some consumer app I'd say go for XML, as it saves you, and the consumer, all that trouble of installing a database.
On modern machines size really doesn't matter.
|
|
|
|
|
Even for a consumer app, it's not so cut and dried.
I'd put the XML / DB cutoff as a table complexity and INSERT / UPDATE / DELETE criteria myself. XML is text based, so frequent changes are going to mean a lot of file copying.
Even for mobile there is SQLCE or SqLite instead, both of which have a relatively small footprint.
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
|
|
|
|
|
For me the question is not quite right.
I don't think row count, or class count should determine whether a database is used or not.
I think it's more a case of what you are going to do with the data.
If you think you may need to 'look' at the data occasionally, manipulate it or run some form of reports(statistics, counts, aggregation...) on the data than a database is the right choice even if you have as few as 100 rows.
If you are just reading in data that has already been 'cleaned' and all you are doing is transforming the data then perhaps a database is not necessary.
As others have mentioned there is no correct answer - whichever path you take there will be advantages and disadvantages.
My rule of thumb is not so much the volume of data, but what is going to happen to the data.
Regarding footprint - there are some databases which leave a reasonably small(suitably vague statement of the day...) footprint, particularly those where the database is structured as one file.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
modified 16-Apr-15 3:35am.
|
|
|
|
|
10,000 is very low number to compare anything for performance.
When you compare XML / Flat file with database the only advantage you get is write performance and cheaper cost. Which is never a deciding factor for any application, whether its enterprise or consumer.
The things you can get with database and not with xml are:
- Concurrency (You can control if you have insert/update at the same time on your data)
- Transaction Reliability (You can go back to an older data/snapshot to correct a mistake)
- Backup (You can configure very efficient backup policy i.e. log shipping / replication which is not possible with XML)
- Partitioning (You can store data to different database/location based on region and then retrieve it easily)
- Archiving (Transfer data to archive in realtime and then access it in realtime with linked database)
- Security (fine grain control over who can access what)
- And the most important, Well Structured and Connected Data (makes complex query for different reports very easy)
So I would suggest you to use XML for intermediate stuff, or to store settings. but other than that don't waste your time and efforts. Just go with the RDBMS.
And for Mobile device to store offline stuff you can use XML / JSON Documents (JSON is better than XML). But at the end you should have a RDBMS server and Sync your offline data to it.
|
|
|
|
|
I'd go for XML with anything that is just a few lines, like configuration. If it contains a lot of relations, I'd switch to SQLite.
If multiple users are to access the datasource, it would also become a real database. I don't want to have to implement things like table-locking.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
If you need any kind of concurrency it can quickly become messy with (xml)files.
|
|
|
|
|
It depends on the structure of the data.
For an example of why this is important, if look at the structure of CodeProject, as you should have before you posted, you will see that you have inserted a Message data item to the Lounge table, which is not where it belongs in the structure.
The effect of this is that stored procedures set up to resolve the type of problem in your Message object will not be able to find it with their pre-configured queries, so your problem will not be found, and will not be resolved.
The world will not end and civilisation will not collapse because of this (probably, but don't quote me if it happens), but it does demonstrate that if the data you have to handle has a complex structure, or many different branches, then storing it in large XML files, e.g. pages of the Lounge, is a pretty damned stupid thing to do.
... Almost as stupid as posting programming questions in the CP Lounge.
I wanna be a eunuchs developer! Pass me a bread knife!
|
|
|
|
|
wreckless wrote: Please enlighten me on the subject.
Data Integrity
Jeremy Falcon
|
|
|
|
|
wreckless wrote: Just kidding about the rock stars
Good, 'cause anyone who thinks he's a rock star is actually just another hipster douchebag.
It's not about size. Others have pointed out some of the many concerns involved.
In short, XML is fine for configuration and reports, but not suitable for random access and multi-user access.
|
|
|
|
|
I knew I should have left the rock stars part out of it. I also learned it's not a good idea to ask a question when I've been drinking. It was just spontaneous combustion I guess. But what the heck, sh#t happens. (Like posting in the lounge, sorry bout that)
Lots of good answers here anyway, It looks to me like if it is simple keep it simple.
If it is complicated, keep it simple by using a database. That about sums it up.
But then again, how do you define simple? What simple for some may be Rock Science to others.
I guess it all depends on what is is?
wreckless
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
