|
Your metrics are too vague to say how much "intelligence" you need but I would consider using Power BI to do a simulation of your requirements to see how much custom code you may (not) need.
I would design the best "operational" system for daily use; and create an "informational" db / extract to help with metrics if needed.
Using Power BI with JSON Data Sources and Files
Power BI Desktop | Microsoft Power BI
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
I can.
But then I have been doing this for a very long time.
From the post as presented I would agree that keeping the data in the object is not a good idea.
|
|
|
|
|
Certainly ... but that is based on "previous" observations and "related" experiences; unless one is in the habit of guessing.
One generally doesn't ask questions on a subject one is already familiar with.
Since the question was asked, I assumed this was a "new" territory for OP. Benchmarking is one of the things I use in new territory.
A sport stats web site thought their 400 MB baseball "game files" were also "too big" for sending over the internet. They compressed down to 25MB when I could have made $ coming up with an alternate solution. Initially, even I thought 400 MB was "big".
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
All I can say is that based on the OP and additional commentary the data that will be stored is going to represent a 'block' that would seem to large as a representation to tie (embed) to that object. Doesn't matter if it is 100k or 100m at the end of the day it is still too 'large'.
And that is in general.
There could be very specific requirements limited to this very specific company and this specific case that might allow it to be there. But that would be an exception not a rule. And even then keeping it separate would not likely be a negative.
So again in general it should not be kept there.
And benchmarking is not actually a consideration here for the initial storage. Rather it is management, ease of use that would initially be my concern. Benchmarking is only applicable to specific business needs and nothing posted so far suggests that is needed.
|
|
|
|
|
It seems the word "benchmarking" is what's causing problems here.
How about: "evaluating your options with quantitative measurements".
Anyways, I'd rather "measure" now than have to explain why performance sucks later.
It seems like we're arguing over who can "guesstimate" (better).
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
"Adding" is cheaper than updating (or inserting) variable-sized chunks. You can "add" two files to become a single file with the DOS copy-command; it would open the first file at EOF, and then start writing the other file.
Vunic wrote: For calcs, Load the whole data in memory, run the aggregate function to parse through the document and produce the metrics.
That's an option, and a reasonable one. I would be running queries on my aggregated DB, mayhaps even using a fancy reporting engine.
Vunic wrote: The document gets updated for the count, when old users make another post.
In the worst case, the first entry needs to be updated (due to an extra character). Means that the rest of the file will be rewritten. If it is 5Mb then this should be quick enough (unless writing on floppies).
Vunic wrote: (Single Document sounds like a Big-Fat approach. I think I'd rather keep them as monthly chunks and make it handy to handle the documents.)
A single document makes life easier for those who need to work with them. Would you prefer big-fat Word documents, or would you prefer Websites as saved by IE? A file, with an accompanying folder and all the files that are linked.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
I think it might be worth pointing out that there is a format where you don't need to play the plain-text document games. You can use BSON constructs to bridge these concerns, such as is used in:
LiteDB - A .NET NoSQL Document Store in a single data file
"There are three kinds of lies: lies, damned lies and statistics."
- Benjamin Disraeli
|
|
|
|
|
Vunic wrote: This document works like an accumulator. For example, lets assume a requirement like this :
I need to store all the users taking part in discussions in CodeProject.
So, I need to capture the User ID & have a counter against each of them.
Your requirements are incomplete.
What happens when a user is deleted?
What happens if the requirements change and now statistics on two different sequences (monthly and weekly) are needed?
This has the current stats but what if someone wants to compare the states from last year to today?
|
|
|
|
|
I have a WPF app. In it there are Companies. Companies can be of different types. For example, there are Builders, Lumber Companies, Electricians, etc. Some of the companies, like Builders, will be loaded into a Navigation bar on app startup. So I need to be able to query companies for those that are builders.
However, One thing I'd like to do is to allow the user to Add/Edit/Remove company types as they see fit. This presents a problem: How do you allow the user to maintain a lookup table of company types, yet in the code determine which are of a certain type?
One possibility is to use the following lookup table structure:
CREATE TABLE Lookups
(
Id INT NOT NULL IDENTITY PRIMARY KEY,
Caption VARCHAR(MAX) NOT NULL,
Category VARCHAR(MAX) NOT NULL,
AppCode VARCHAR(MAX) NULL,
[Description] VARCHAR(MAX),
IsSystem BIT
)
If a Company Type is required for the app to run, then set IsSystem to true and provide an AppCode like "company_type_builder".
This would allow me to query for companies that have their type set to that Lookup Id. The lookup types that are set as IsSystem cannot be removed by the user.
If I decide not to use a lookup table and make the company type fixed in code, then an enum would suffice.
What are your thoughts on this?
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
Ya can't fix stupid.
modified 28-Nov-17 12:57pm.
|
|
|
|
|
If the user needs to add/edit/remove company types, then you have to use a lookup table. An enum is hard-coded, and whilst it can store values that aren't defined, they will only ever show as numbers.
Rather than having undeletable types, why not have a flag indicating whether the type should be shown on the navigation bar? That way, you give the user more control over the application.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
I'm curious, do you use varchar(MAX) for every text field?
And I can't improve on Richards answer 
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
In this case yes. Once I get further into the design I may make the field sizes more fixed. For now I don't really know how big these fields need to be.
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
Ya can't fix stupid.
|
|
|
|
|
I studied the design patterns book written by the Gang of Four when it first hit the shelves in the mid 1990's and I have referred to it and used patterns on only a few occasions since then. It never really caught on with me and became an automatic go-to while I'm at the design stage.
Do you use design patterns on a regular basis?
In your view, has it fallen out of favor in the industry?
If you use patterns regularly, do you have a source or sources of new patterns?
TIA
Cheers,
Mike Fidler
"I intend to live forever - so far, so good." Steven Wright
"I almost had a psychic girlfriend but she left me before we met." Also Steven Wright
"I'm addicted to placebos. I could quit, but it wouldn't matter." Steven Wright yet again.
|
|
|
|
|
MikeTheFid wrote: Do you use design patterns on a regular basis?
Yes, and you probably do too. People tend to forget that a design pattern is just a formal name and description of a solution to a problem that crops up frequently. It's a convenient shortcut for conveying a solution to the problem; it's quicker to say "use a factory" than to describe how a factory works - but ultimately you would end up writing the same code.MikeTheFid wrote: In your view, has it fallen out of favor in the industry?
Nope. As evidence, I would point to Angular (which uses MVVM heavily) and ASP MVC (the clue is in the name).MikeTheFid wrote: If you use patterns regularly, do you have a source or sources of new patterns?
Code Project, GoF, articles, peers and many, many other sources.
This space for rent
|
|
|
|
|
MikeTheFid wrote: In your view, has it fallen out of favor in the industry?
You'll find patterns throughout the .NET framework, from strategy to decorators.
MikeTheFid wrote: If you use patterns regularly, do you have a source or sources of new patterns?
There is no single authority, so you'd always have multiple sources. If you have the patience, then YouTube is an option. I prefer books, like Manning | SOA Patterns[^] 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
The "patterns" are out there and are being used; whether the "users" realize it or not.
What did NOT happen, was that we'd all be sitting around "talking patterns" (like architects talking about "flying butresses" and the like).
The odds of finding 2 people in the same room who CAN talk patterns is zero.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: The odds of finding 2 people in the same room who CAN talk patterns is zero.
I would not state that during an interview
At Cadac (previous employer) there's an architect that will take over the whiteboard and start explaining how the group of devs is going to build an application; it is assumed that everyone is at least up to date on the most used patterns. You're a developer, after all, and the factory, singleton or decorator aren't a new idea.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
That was ONE person doing the "talking" ... "previously".
And most "architects" rarely "implement"; they usually impede.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: That was ONE person doing the "talking" ... "previously".
The way you phrase that so carefully makes me feel uneasy, and I have no clue why 
Gerry Schmitz wrote:
And most "architects" rarely "implement"; they usually impede.
That's the task of the manager, not the architect.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Eddy Vluggen wrote: You're a developer, after all, and the factory, singleton or decorator aren't a new idea.
Yes, but does that really relate to what GoF defines?
Following is the GoF definition for "Factory Method" (there is no "Factory").
"Define an interface for creating an object, but let subclasses decide which class to instantiate. Factory Method lets a class defer instantiations to subclasses."
Lets look at "Abstract Factory". The GoF definition.
"Provide an interface for creating families of related or dependent objects without specifying their concrete classes."
When you and/or the Architect says 'Factory' in a design discussion is the idea that it will follow, exactly, the first definition above?
If it helps note that the GoF also says the following about a "Factory Method"
Use the Factory Method pattern when
- a class can't anticipate the class of objects it must create
- a class wants its subclasses to specify the objects it creates
- classes delegate responsibility to one of several helper subclasses and you want to localize the knowledge of which helper subclass is the delegate.
As for myself I am rather certain I follow neither. I expect that I use "Factory" to mean what GoF documents in the "Factory Method" to be 'ConcreteCreator' and I might use "Abstract Factory" in implementation (if I use it at all) as something that more closely resembles what "Factory Method" is. But not exactly.
|
|
|
|
|
jschell wrote:
Yes, but does that really relate to what GoF defines?
I'd seriously wouldn't know, I don't go there often
jschell wrote:
When you and/or the Architect says 'Factory' in a design discussion is the idea that it will follow, exactly, the first definition above?
No, just the generic idea, otherwise he would specify exactly what the requirements are. If he doesn't, then it is up to the developer to choose the most appropriate solution - which usually means going for the simplest thing possible (whilst still only returning an interface).
..but when in doubt, simply ask 
jschell wrote: As for myself I am rather certain I follow neither. I expect that I use "Factory" to mean what GoF documents in the "Factory Method" to be 'ConcreteCreator' and I might use "Abstract Factory" in implementation (if I use it at all) as something that more closely resembles what "Factory Method" is. But not exactly.
Depends heavily on what is required; do you want a static factory, a singleton, or would you prefer an object? If it has to be combined with an object-manager pattern, then I'd think that the object is preferred. Do you want one object, or do you want multiple objects? In the first case, all objects created in the factory will be available over the manager globally. In the second case, only those objects are available that the specific factory created (in that thread).
Point is, you don't want to explain how a "single point of creation in code" can save you from having to update a 1000 references in code where there's normally a "new"-keyword. That's the problem that a factory solves, with the trade-off that one introduces a tiny bit of overhead (in terms of execution).
It is not just something you learn to impress during the interview, they're simply descriptions of a way to solve something. Lots of "developers" here would struggle with a undo/redo pattern, while others simply request a memento-pattern.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
All true, but in terms of the OP the specifics of what is in GoF doesn't seem to really have been all that useful useful.
|
|
|
|
|
Seems like a collection of formalized descriptions of some common patterns, but without much explanation.
I don't think the website is meant as a source to learn those patterns; that is what their book is for
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Eddy Vluggen wrote: Seems like a collection of formalized descriptions of some common patterns, but without much explanation.
Ok, but I was actually referencing the book. The copy of the book sits on my shelf above my computer. Been there for years.
|
|
|
|
|
Quote: The "patterns" are out there and are being used; whether the "users" realize it or not.
Patterns like the "copy/paste"- pattern and the "do it fast and Dirty"- pattern 
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
