|
honey the codewitch wrote: Rather than doing JOINs in the database he was doing them in PHP on the webserver
I had a colleague like that once. He was transferring two 100G entries tables (+-50 columns each) to PHP to do joins using for loops. It was killing the server and taking almost a week to process.
The boss only noticed when he requested a new, more powerful server (our server was brand new). Boss told me to converted it to SQL and it started taking less than an hour to do the same join
That colleague quit the project soon after that.
|
|
|
|
|
ElectronProgrammer wrote: That colleague quit the project soon after that.
Did he fall, or was he pushed?
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
He fell. It wasn't his only mistake. I tried to give him an hand but he took the arm to harm (he had a PhD and I had a licenciate degree*). I would rather have him there than ending overworked and underpaid as I did. I guess the corporate reality train was to heavy for him
* That degree is one below masters and one above bachelor in pre-bologna process which was my case
|
|
|
|
|
About five+ years back I normalized the hell out of databases, but I've come to realize a surprising amount of data needs to be stored redunantly anyway.
Stuff like customer names and addresses on invoices, sales people's names on orders, etc.
In the age of microservices every service needs to have the data it needs to do its thing.
So I now even have multiple databases with the same data because multiple services operate on that data.
Although I usually normalize my services as well...
|
|
|
|
|
That's true, and given the amount of space to work with that we have these days, and how good databases are at doing batch data conversion and such, there really isn't a good reason to avoid redundant data if it makes things easier or more efficient. You just have to be careful not to go crazy with it, and you have to keep in mind that it can make your database more "brittle" because its more fields that need to be proofed/validated for correctness. Keeping redundant data in sync is a chore unto itself.
But yes yes yes to this. Redundant data is okay, when it serves a purpose, often for performance or integration purposes it is The Right Way(TM) to do things.
And if I'm wrong then I got paid a whole lot of money to be wrong, and a lot of people happy with how wrong I was.
Real programmers use butterflies
|
|
|
|
|
honey the codewitch wrote: The Right Way(TM) to do things. Math.
honey the codewitch wrote: And if I'm wrong then I got paid a whole lot of money to be wrong Yeah. Happy my surgeon isn't like that
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Eddy Vluggen wrote: Math. NoMATH (for clarity, Not Only Math)
Actually, that math was failing with big enough datasets, which is why NoSQL was (re-)invented.
So the right way to do things went from "normalize everything" to "denormalize a good bunch."
We've seen the opposite happen to peanut butter.
|
|
|
|
|
I find No SQL kind of funny as it is one of those cyclic patterns.
In the same vein with centralized versus distributed.
|
|
|
|
|
Before SQL, which didn't become popular until the 80's, there was only not SQL
|
|
|
|
|
But don't forget what Edsger Dijstra once said: If you say the same thing twice, you will contradict yourself.
|
|
|
|
|
Not if you say it properly. *hides*
Real programmers use butterflies
|
|
|
|
|
When you're talking about redundant storage, are you referring to data aggregated from single points of origin or an uncontrolled mess?
|
|
|
|
|
Why not both?
But seriously, what starts out as the first sometimes turns into the latter
Mostly the first though
|
|
|
|
|
And some redundancy is not really redundant, it is the historical context that needs to be saved.
The invoice earlier is a good example.
If the company changes its name or moves it’s HQ, you do not want the invoice to update.
The best part is when there are M/A and divestitures among your client base.
At a high enough altitude, it looks like a game of Life cellular generational display.
|
|
|
|
|
Sander Rossel wrote: In the age of microservices every service needs to have the data it needs to do its thing. Good point, those use a smaller dataset.
Sander Rossel wrote: So I now even have multiple databases with the same data because multiple services operate on that data. Operating implies you doin' more than reading that data.
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Eddy Vluggen wrote: Operating implies you doin' more than reading that data. It's mostly reading and transforming.
Not like, changing the data and then posting it back to its origin or something like that, that would be chaos.
|
|
|
|
|
What if my databases are abnormal?
|
|
|
|
|
So it WAS YOU!
You are the reason I've made more money migrating away from nightmare data than writing new code.
Real programmers use butterflies
|
|
|
|
|
Your database, not mine. Do as you like
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Is your name Marty Feldman?
|
|
|
|
|
Usually, and usually just down to 3NF, but with "an eye" to avoid going too far or going that step further.
But very rarely these days do I get to do that sort of stuff - I'm stuck with what was someone else's idea of Good Database Structure. We disagree on what that looks like.
|
|
|
|
|
CHill60 wrote: I'm stuck with what was someone else's idea of Good Database Structure That's why we normalize.
One of my "betters" once introduced a table that held a varchar(31) with 0's and 1's. Representing booleans. Queries where horrid and slow.
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Eddy Vluggen wrote: ntroduced a table that held a varchar(31) with 0's and 1's.
I had a colleague who used 1s and 8s -- because he couldn't get 0s to work correctly.
modified 1-Nov-21 13:57pm.
|
|
|
|
|
This is why fights start in the office.
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
I would if I could.
In fact, I do as much as I can, which is limited and not always beneficial.
But the issue is that my current primary task is simply ETLing data from various sources to our staging database. For that, it's usually best to just leave it as is. But, at times, I do split some data into child tables -- usually if the incoming data has been de-normalized such that a field contains delimited lists of values (yuck).
It really depends on the needs of the next layer of the overall application, which often requires de-normalized data anyway.
On those rare occasions when I create a database for a "proper" application, I normalize as required for the application.
But I don't care about the definitions of the various normal forms; I leave that to the ivory tower types.
|
|
|
|