|
Vunic wrote: On the other hand, if I'm distributing it into multiple documents, it will need proper indexing/partitionin/sharding things done right.
JSON, XML and CSV are great formats for exporting data; but they do not work well as datastores where frequent updates are expected. If your data contains images, they'll probably be stored in Base64, making the file even larger. If there is no updates expected to the data, and it is merely a static export - then one of the text-based exports is preferred. Reading is fast enough for text, but updating takes a severe and noticable penalty.
Vunic wrote: Though a single document looks neat on the design
Also a lot easier when making backups and sending stuff. I prefer to put all data in a single file; with a file-based database like SQLite, SQLCe and MSAccess you can easily even store binary data. Another nicety is that you can link to these databases from SQL Server and interact with them as if they were local SQL Server databases.
If the data is not related at all, then I'd recommend zipping your various documents, so it remains "one file" in the eye of the end-user. Since you initially chose JSON, I'd be expecting data that is somehow related.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
I have updated the OP[^] Please check the sample scenario.
Starting to think people post kid pics in their profiles because that was the last time they were cute - Jeremy Falcon.
|
|
|
|
|
You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
Most internet traffic is automatically "zipped" (gzipped) by default; and will on average compress to 10% of its original (text) size. That addresses bandwidth.
Where else is is it "too big"?
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
If you look at my post, that's exactly what I did. Instead of benchmarking, I'll remind you that any text-based file needs to be rewritten to update, and it does not support random access.
Zipping such a text-file means rewriting and rezipping the entire stream, and writing that result to disc on every change. That's fine for a datadump in CSV that is imported, but it is not a ideal structure for data that tends to be modified.
Gerry Schmitz wrote: Where else is is it "too big"?
GMail, if the attachment is larger than 25Mb. So yes, would depend on context, wouldn't it? 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
I wasn't talking to you.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
No, you were posting on a public forum which allows answering even if someone is not talking to you. Legally and technically no problem, and not intended to offend.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Except you usually miss the point / "context" (as in this case) and I have no interest in "debating it" when I wasn't addressing you.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry, you're normally good when answering but in this case, you really are coming across as a jerk. Rather than defending your position, you have made a blanket statement and are being hostile when presented with opinions that you don't like. Eddy presented situations where it's "too big" and you're ignoring him just because he's not the OP; oh, and your point was only a very minor part of what the OP might have wanted to do so you were making assumptions as to the intended purpose, and failed to address the larger issues, which Eddy did.
This space for rent
|
|
|
|
|
I was asking OP to elaborate: in what "context" did he think "his" data was too big. My experience re: JSON was in REST; I was alluding to bandwidth. i.e. the internet. And that browsers generally "zip" unless you turn it off.
Instead, I get addressed as if "I" was questioning said "offendee" about "his" benchmarking? And then he goes off on "email" attachments?
If one waits "a round or two", the context becomes more clear, and you don't come off as a "jerk" by derailing someone else because you're so eager to show "how smart your are".
It's (his) bait; you just can't see it but took it.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: If one waits "a round or two", the context becomes more clear, and you don't come off as a "jerk" by derailing someone else because you're so eager to show "how smart your are".
The hardest part is to get to the first round. I did not intend to be a jerk, but don't mind to be called that either
Gerry Schmitz wrote: It's (his) bait; you just can't see it but took it.
FWIW; there may come a day when we are forced to work on the same team.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
I was being rhetorical at this point because Hanlon has already said that I'm the "jerk" here.
And I wouldn't "force" anyone or myself to work with anyone they didn't want to. Or is that what you're used to?
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: I was being rhetorical at this point because Hanlon has already said that I'm the "jerk" here.
That's one opinion; enough people will say that I am a well trained jerk. Luckily it is not a competition.
Gerry Schmitz wrote: And I wouldn't "force" anyone or myself to work with anyone they didn't want to. Or is that what you're used to?
I prefer to work with people who are honest and who aren't afraid to say I might be wrong. Since you're a good programmer (with posts here to back that up) I'd say you fit the profile perfectly 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
To my users, I'm a necessary evil.
(Supposedly, those who "swear" a lot are also more "honest").
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: To my users, I'm a necessary evil.
My co workers may same they same thing. We do this because we're good with computers, if we were good with humans we would be doing something entirely different.
Just published an article on CP on the subject of patterns, and given the thread on patterns below, I would like your opinion. Would be worth a lot if it would change your initial reaction to "some patterns could be usefull" 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
If we were good with humans, we would be rich.
Never said patterns weren't usefull ... said most conversations (in my experience) aren't at a level where more than one person can "talk" "GO4" patterns. (and "most" think of GO4 if they think patterns; not someone else's).
All my "patterns" are based around "adapters" and builders and repositories: devices; threads; monitors; charts; recorders; serializers; 3rd party API; etc.
"Facades" and "decorators" just don't enter the conversation.
And my patterns evolve; some patters (industrial) I have modified over time based on "looking" at it in a different way.
New patterns and algorithms: quantum computing ...
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: Never said patterns weren't usefull ... said most conversations (in my experience) aren't at a level where more than one person can "talk" "GO4" patterns. (and "most" think of GO4 if they think patterns; not someone else's).
That must be some good marketing from the GoF.
Gerry Schmitz wrote: If we were good with humans, we would be rich.
..or in prison for attempting 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Good marketing? Just a testimony to how slavish some organizations can be; no independent thought.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
|
Your metrics are too vague to say how much "intelligence" you need but I would consider using Power BI to do a simulation of your requirements to see how much custom code you may (not) need.
I would design the best "operational" system for daily use; and create an "informational" db / extract to help with metrics if needed.
Using Power BI with JSON Data Sources and Files
Power BI Desktop | Microsoft Power BI
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
I can.
But then I have been doing this for a very long time.
From the post as presented I would agree that keeping the data in the object is not a good idea.
|
|
|
|
|
Certainly ... but that is based on "previous" observations and "related" experiences; unless one is in the habit of guessing.
One generally doesn't ask questions on a subject one is already familiar with.
Since the question was asked, I assumed this was a "new" territory for OP. Benchmarking is one of the things I use in new territory.
A sport stats web site thought their 400 MB baseball "game files" were also "too big" for sending over the internet. They compressed down to 25MB when I could have made $ coming up with an alternate solution. Initially, even I thought 400 MB was "big".
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
All I can say is that based on the OP and additional commentary the data that will be stored is going to represent a 'block' that would seem to large as a representation to tie (embed) to that object. Doesn't matter if it is 100k or 100m at the end of the day it is still too 'large'.
And that is in general.
There could be very specific requirements limited to this very specific company and this specific case that might allow it to be there. But that would be an exception not a rule. And even then keeping it separate would not likely be a negative.
So again in general it should not be kept there.
And benchmarking is not actually a consideration here for the initial storage. Rather it is management, ease of use that would initially be my concern. Benchmarking is only applicable to specific business needs and nothing posted so far suggests that is needed.
|
|
|
|
|
It seems the word "benchmarking" is what's causing problems here.
How about: "evaluating your options with quantitative measurements".
Anyways, I'd rather "measure" now than have to explain why performance sucks later.
It seems like we're arguing over who can "guesstimate" (better).
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
"Adding" is cheaper than updating (or inserting) variable-sized chunks. You can "add" two files to become a single file with the DOS copy-command; it would open the first file at EOF, and then start writing the other file.
Vunic wrote: For calcs, Load the whole data in memory, run the aggregate function to parse through the document and produce the metrics.
That's an option, and a reasonable one. I would be running queries on my aggregated DB, mayhaps even using a fancy reporting engine.
Vunic wrote: The document gets updated for the count, when old users make another post.
In the worst case, the first entry needs to be updated (due to an extra character). Means that the rest of the file will be rewritten. If it is 5Mb then this should be quick enough (unless writing on floppies).
Vunic wrote: (Single Document sounds like a Big-Fat approach. I think I'd rather keep them as monthly chunks and make it handy to handle the documents.)
A single document makes life easier for those who need to work with them. Would you prefer big-fat Word documents, or would you prefer Websites as saved by IE? A file, with an accompanying folder and all the files that are linked.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
I think it might be worth pointing out that there is a format where you don't need to play the plain-text document games. You can use BSON constructs to bridge these concerns, such as is used in:
LiteDB - A .NET NoSQL Document Store in a single data file
"There are three kinds of lies: lies, damned lies and statistics."
- Benjamin Disraeli
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
