|
Gerry, you're normally good when answering but in this case, you really are coming across as a jerk. Rather than defending your position, you have made a blanket statement and are being hostile when presented with opinions that you don't like. Eddy presented situations where it's "too big" and you're ignoring him just because he's not the OP; oh, and your point was only a very minor part of what the OP might have wanted to do so you were making assumptions as to the intended purpose, and failed to address the larger issues, which Eddy did.
This space for rent
|
|
|
|
|
I was asking OP to elaborate: in what "context" did he think "his" data was too big. My experience re: JSON was in REST; I was alluding to bandwidth. i.e. the internet. And that browsers generally "zip" unless you turn it off.
Instead, I get addressed as if "I" was questioning said "offendee" about "his" benchmarking? And then he goes off on "email" attachments?
If one waits "a round or two", the context becomes more clear, and you don't come off as a "jerk" by derailing someone else because you're so eager to show "how smart your are".
It's (his) bait; you just can't see it but took it.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: If one waits "a round or two", the context becomes more clear, and you don't come off as a "jerk" by derailing someone else because you're so eager to show "how smart your are".
The hardest part is to get to the first round. I did not intend to be a jerk, but don't mind to be called that either 
Gerry Schmitz wrote: It's (his) bait; you just can't see it but took it.
FWIW; there may come a day when we are forced to work on the same team.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
I was being rhetorical at this point because Hanlon has already said that I'm the "jerk" here.
And I wouldn't "force" anyone or myself to work with anyone they didn't want to. Or is that what you're used to?
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: I was being rhetorical at this point because Hanlon has already said that I'm the "jerk" here.
That's one opinion; enough people will say that I am a well trained jerk. Luckily it is not a competition.
Gerry Schmitz wrote: And I wouldn't "force" anyone or myself to work with anyone they didn't want to. Or is that what you're used to?
I prefer to work with people who are honest and who aren't afraid to say I might be wrong. Since you're a good programmer (with posts here to back that up) I'd say you fit the profile perfectly 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
To my users, I'm a necessary evil.
(Supposedly, those who "swear" a lot are also more "honest").
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: To my users, I'm a necessary evil.
My co workers may same they same thing. We do this because we're good with computers, if we were good with humans we would be doing something entirely different.
Just published an article on CP on the subject of patterns, and given the thread on patterns below, I would like your opinion. Would be worth a lot if it would change your initial reaction to "some patterns could be usefull" 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
If we were good with humans, we would be rich.
Never said patterns weren't usefull ... said most conversations (in my experience) aren't at a level where more than one person can "talk" "GO4" patterns. (and "most" think of GO4 if they think patterns; not someone else's).
All my "patterns" are based around "adapters" and builders and repositories: devices; threads; monitors; charts; recorders; serializers; 3rd party API; etc.
"Facades" and "decorators" just don't enter the conversation.
And my patterns evolve; some patters (industrial) I have modified over time based on "looking" at it in a different way.
New patterns and algorithms: quantum computing ...
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: Never said patterns weren't usefull ... said most conversations (in my experience) aren't at a level where more than one person can "talk" "GO4" patterns. (and "most" think of GO4 if they think patterns; not someone else's).
That must be some good marketing from the GoF.
Gerry Schmitz wrote: If we were good with humans, we would be rich.
..or in prison for attempting 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Good marketing? Just a testimony to how slavish some organizations can be; no independent thought.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
|
Your metrics are too vague to say how much "intelligence" you need but I would consider using Power BI to do a simulation of your requirements to see how much custom code you may (not) need.
I would design the best "operational" system for daily use; and create an "informational" db / extract to help with metrics if needed.
Using Power BI with JSON Data Sources and Files
Power BI Desktop | Microsoft Power BI
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
I can.
But then I have been doing this for a very long time.
From the post as presented I would agree that keeping the data in the object is not a good idea.
|
|
|
|
|
Certainly ... but that is based on "previous" observations and "related" experiences; unless one is in the habit of guessing.
One generally doesn't ask questions on a subject one is already familiar with.
Since the question was asked, I assumed this was a "new" territory for OP. Benchmarking is one of the things I use in new territory.
A sport stats web site thought their 400 MB baseball "game files" were also "too big" for sending over the internet. They compressed down to 25MB when I could have made $ coming up with an alternate solution. Initially, even I thought 400 MB was "big".
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
All I can say is that based on the OP and additional commentary the data that will be stored is going to represent a 'block' that would seem to large as a representation to tie (embed) to that object. Doesn't matter if it is 100k or 100m at the end of the day it is still too 'large'.
And that is in general.
There could be very specific requirements limited to this very specific company and this specific case that might allow it to be there. But that would be an exception not a rule. And even then keeping it separate would not likely be a negative.
So again in general it should not be kept there.
And benchmarking is not actually a consideration here for the initial storage. Rather it is management, ease of use that would initially be my concern. Benchmarking is only applicable to specific business needs and nothing posted so far suggests that is needed.
|
|
|
|
|
It seems the word "benchmarking" is what's causing problems here.
How about: "evaluating your options with quantitative measurements".
Anyways, I'd rather "measure" now than have to explain why performance sucks later.
It seems like we're arguing over who can "guesstimate" (better).
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
"Adding" is cheaper than updating (or inserting) variable-sized chunks. You can "add" two files to become a single file with the DOS copy-command; it would open the first file at EOF, and then start writing the other file.
Vunic wrote: For calcs, Load the whole data in memory, run the aggregate function to parse through the document and produce the metrics.
That's an option, and a reasonable one. I would be running queries on my aggregated DB, mayhaps even using a fancy reporting engine.
Vunic wrote: The document gets updated for the count, when old users make another post.
In the worst case, the first entry needs to be updated (due to an extra character). Means that the rest of the file will be rewritten. If it is 5Mb then this should be quick enough (unless writing on floppies).
Vunic wrote: (Single Document sounds like a Big-Fat approach. I think I'd rather keep them as monthly chunks and make it handy to handle the documents.)
A single document makes life easier for those who need to work with them. Would you prefer big-fat Word documents, or would you prefer Websites as saved by IE? A file, with an accompanying folder and all the files that are linked.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
I think it might be worth pointing out that there is a format where you don't need to play the plain-text document games. You can use BSON constructs to bridge these concerns, such as is used in:
LiteDB - A .NET NoSQL Document Store in a single data file
"There are three kinds of lies: lies, damned lies and statistics."
- Benjamin Disraeli
|
|
|
|
|
Vunic wrote: This document works like an accumulator. For example, lets assume a requirement like this :
I need to store all the users taking part in discussions in CodeProject.
So, I need to capture the User ID & have a counter against each of them.
Your requirements are incomplete.
What happens when a user is deleted?
What happens if the requirements change and now statistics on two different sequences (monthly and weekly) are needed?
This has the current stats but what if someone wants to compare the states from last year to today?
|
|
|
|
|
I have a WPF app. In it there are Companies. Companies can be of different types. For example, there are Builders, Lumber Companies, Electricians, etc. Some of the companies, like Builders, will be loaded into a Navigation bar on app startup. So I need to be able to query companies for those that are builders.
However, One thing I'd like to do is to allow the user to Add/Edit/Remove company types as they see fit. This presents a problem: How do you allow the user to maintain a lookup table of company types, yet in the code determine which are of a certain type?
One possibility is to use the following lookup table structure:
CREATE TABLE Lookups
(
Id INT NOT NULL IDENTITY PRIMARY KEY,
Caption VARCHAR(MAX) NOT NULL,
Category VARCHAR(MAX) NOT NULL,
AppCode VARCHAR(MAX) NULL,
[Description] VARCHAR(MAX),
IsSystem BIT
)
If a Company Type is required for the app to run, then set IsSystem to true and provide an AppCode like "company_type_builder".
This would allow me to query for companies that have their type set to that Lookup Id. The lookup types that are set as IsSystem cannot be removed by the user.
If I decide not to use a lookup table and make the company type fixed in code, then an enum would suffice.
What are your thoughts on this?
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
Ya can't fix stupid.
modified 28-Nov-17 12:57pm.
|
|
|
|
|
If the user needs to add/edit/remove company types, then you have to use a lookup table. An enum is hard-coded, and whilst it can store values that aren't defined, they will only ever show as numbers.
Rather than having undeletable types, why not have a flag indicating whether the type should be shown on the navigation bar? That way, you give the user more control over the application.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
I'm curious, do you use varchar(MAX) for every text field?
And I can't improve on Richards answer 
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
In this case yes. Once I get further into the design I may make the field sizes more fixed. For now I don't really know how big these fields need to be.
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
Ya can't fix stupid.
|
|
|
|
|
I studied the design patterns book written by the Gang of Four when it first hit the shelves in the mid 1990's and I have referred to it and used patterns on only a few occasions since then. It never really caught on with me and became an automatic go-to while I'm at the design stage.
Do you use design patterns on a regular basis?
In your view, has it fallen out of favor in the industry?
If you use patterns regularly, do you have a source or sources of new patterns?
TIA
Cheers,
Mike Fidler
"I intend to live forever - so far, so good." Steven Wright
"I almost had a psychic girlfriend but she left me before we met." Also Steven Wright
"I'm addicted to placebos. I could quit, but it wouldn't matter." Steven Wright yet again.
|
|
|
|
|
MikeTheFid wrote: Do you use design patterns on a regular basis?
Yes, and you probably do too. People tend to forget that a design pattern is just a formal name and description of a solution to a problem that crops up frequently. It's a convenient shortcut for conveying a solution to the problem; it's quicker to say "use a factory" than to describe how a factory works - but ultimately you would end up writing the same code.MikeTheFid wrote: In your view, has it fallen out of favor in the industry?
Nope. As evidence, I would point to Angular (which uses MVVM heavily) and ASP MVC (the clue is in the name).MikeTheFid wrote: If you use patterns regularly, do you have a source or sources of new patterns?
Code Project, GoF, articles, peers and many, many other sources.
This space for rent
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
