|
honey the codewitch wrote: He doesn't run servers out of his home anymore. Maybe he just shouldn't run server anymore, period
|
|
|
|
|
Correct.
To err is human. Fortune favors the monsters.
|
|
|
|
|
I entered kill 0 (instead of kill 9) on a SunOS server one time and it failed to come back up. The problem is it was a core server for a large number of administrative people. Turns out it had previously corrupted the file system. It was a week before it was available again.
|
|
|
|
|
Also decades ago I worked for a rather large broking company. Sysadmin could not work out who owned a particular server (there were a lot of servers) so he shut it down to see who screamed. It turned out to be a company that had been sold off 4 years before and moved 2 floors down.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
Ah, the 'ol scream test. An oldie but a goodie. But, potentially career limiting if the wrong person screams.
|
|
|
|
|
I've done it more than once. The screams usually come pretty quickly.
|
|
|
|
|
I've had to do this as well since no one seems to know what their resources are.
|
|
|
|
|
One chap I was called to help around 25 years ago explained to me that every time he pressed a key on his keyboard his computer would restart.
So I went over to his desk and observed what was happening and yes the computer did shut down every time he pressed a key on the keyboard.
Now my Sherlock Holmes senses kicked in as I noticed that he also had a folder resting on his lap.
Every time he leaned forward to press a key, on the keyboard, the corner of the folder pressed the restart button on his computer...
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
modified 26-Aug-22 3:43am.
|
|
|
|
|
I once deleted ALL of the car loans for FL because someone changed the "mechanical" terminal Switch,
and I was NO LONGER on the DEV box.
OUCH!
A month later, someone else did a different state. And I patched the operating systems for production to block the command (renamed it to another command we would NEVER type accidentally)...
|
|
|
|
|
|
|
I accidently tested a database shutdown script I was developing on the PROD system, but who hasn't done that?
One day I found out that a DEC AlphaServer 1000 has a switch which kills power if the case is opened while the system is running. I mean, WTF? Definitely over-engineered. All I was doing was determining whether or not there were any spare RAM slots. : sheesh : At least that was a DEV system.
|
|
|
|
|
Hmm,
Done both of those! 8)
Worst for me was - on a production system - deleting an 'old' copy of a database from the backup filestore during a very busy morning. You guessed it - not the 'old' copy at all! This was all long before SQL (or even Linux) was a thing on PCs.
Had to restore from the overnight tape backup (we did have backups, and they were regularly tested, fortunately) and then get the reservation staff to go through all the paper printouts of that day's bookings to recover the mornings transactions, whilst suspending new ones to guard against double bookings. (Fortunately I also produced a paper copy of every transaction as - back in the day - neither the power nor tape backups were 100% reliable).
I was not Mr popular that day, I can tell you!
I learnt my lesson though and added an hourly local backup to a .zip file of transaction data just in case. That saved my bacon a few times too!
|
|
|
|
|
|
What the !?
Was this guy fired?
Did he die in a freak "accident"?
This is closer to sabotage than to ignorance
|
|
|
|
|
Sander Rossel wrote: Was this guy fired? Nope. The guy who did it, just did what he was ordered to.
The explanation is: big OEM company.
It is not that the right hand doesn't know what the left hand is doing. There are many times when not even the fingers of the same hand know what the others are doing.
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
Back in the mid-80’s at a former employer who did photo processing they did a major upgrade of the DEC PDO-11/70 systems located in their multiple film processing plants to Microvax. Each of the new Microvax systems was setup with two large hard drives with one being mirrored on the other. They were hot swappable so if something went wrong with one drive they could stop it and keep going with the other one while the bad drive was sent for repairs. So one morning at work we were alerted to a problem that a bunch of invoices were missing in one of the plants. It turned out that at this particular plant they had to get a hard drive repaired and when it came back a few days later they started mirroring the drives again only someone (not me, thankfully) did the mirroring in the wrong direction so the drive that had the old data overwrote the drive with the current data. I’m not sure what they ultimately did to get the data back since I don’t think there was a tape backup because of the disk mirroring (I think it was probably the same person who thought they no longer needed the redundant DECNet cross links between the plants) but I’m thinking one of the devs had to write something so some poor workers in the plant could do a bunch of data entry.
|
|
|
|
|
Classic!
Actually iTunes did that to my iPod once.
First (and last) time I used it.
Don't even own an iPod anymore now.
No Apple a day keeps the doctor away
|
|
|
|
|
Congrats on your immortality! And welcome to the club
|
|
|
|
|
I was immortalized many years ago actually
You don't get to be in IT for twelve years without accidentally deleting a table or two in production (well, I never did that, but locking an entire database was pretty bad too)
|
|
|
|
|
About 25 years ago I learned an axiom that I live by.
"It only takes one "Oh Sh*t!" to wipe out 25 "Atta boy"s.
|
|
|
|
|
Good post. There is absolutely no point working when you're tired. You get next to nothing useful done and you make mistakes. It also leads to burnout, which can be devastating. It (burnout) happened to me and also to a friend of mine, and it put us both in hospital (at different times). It took me a long time to figure all this out, but I've got it now.
I work for myself nowadays so I can do what I like, but more employers need to understand this and to take proper care of the mental health of their staff.
Sorry Sander, maybe a bit more than you were expecting, but I feel very strongly about this.
PS: Take breaks. Take breaks, take breaks, take breaks.
Paul Sanders.
If I had more time, I would have written a shorter letter - Blaise Pascal.
Some of my best work is in the undo buffer.
|
|
|
|
|
My first job was 100% Assembler. In an effort to squeeze out a few more bytes of memory for variables, I was cleaning up a program and deleted a single space located before the variable used in the print command. The program worked fine but, printed a few boxes of pre-printed forms with garbage due to the preceding valid blank space being removed leaving another character to be mvcl'd across the print line before it was built. Almost a career limiting move.
|
|
|
|
|
Hard work, assembler. I'm too old now to have the focus you need (but there are compensations).
My favourite oh no! story is from my college days, programming on punch cards (ouch, that really dates me). Somehow, a rogue card sneaked its way into the deck and as a result my printout consisted of 2000 or so blank pages. I bet the ops loved that!
Paul Sanders.
If I had more time, I would have written a shorter letter - Blaise Pascal.
Some of my best work is in the undo buffer.
|
|
|
|
|
Paul Sanders (the other one) wrote: There is absolutely no point working when you're tired. I sometimes get into a work vibe when I stumble across a problem and I make it personal.
Kind of an adrenaline rush I guess
Paul Sanders (the other one) wrote: Sorry Sander, maybe a bit more than you were expecting, but I feel very strongly about this. I agree wholeheartedly though
Paul Sanders (the other one) wrote: PS: Take breaks. Take breaks, take breaks, take breaks. I take works from my break
|
|
|
|