I work full-time as an IT manager, and once in a while, I accept little contracts for SMBs when the job is short and the money is good.
About a year ago, I was contacted by "Greg," the computer guy in a construction company, for help migrating a server to VMware. Greg had been given my name by the company's accountant, who had once been a summer student with a company I used to work for. (I found out later that the new server project was the accountant's initiative, and he was not comfortable with Greg's skills.) Since Greg was alone doing IT at the construction company, it was an administrator position even if his knowledge and experience was that of a first-level technician.
[ Want to cash in on your IT experiences? Send your story to offtherecord@infoworld.com. If we publish it, we'll send you a $50 American Express gift card. | Show off your geek quotient: Get your InfoWorld shirts, cups, and more schwag. ]
Greg had a powerful server that was used for accounting and wanted to convert its existing OS to a virtual machine and run new virtual servers on it. Because he didn't know Linux or VMware, he asked me to come over and reinstall his server for him. So around 9 a.m. on a Saturday morning, we got started and outlined our plan of attack. I advised him to do a physical-to-virtual conversion, but he preferred to reinstall the server from scratch with the accounting application and restore the database later.
Before starting I asked him to perform backups of his data to make sure we had everything. He said that wasn't necessary, that the backup was likely to have run last night, and he showed me his tapes. I insisted that likely wasn't good enough in IT and that we should do a backup right away to be safe. I guess he was worried about my hourly rate because he refused.
I reluctantly formatted the server and performed a basic Debian install, then installed VMware server. Greg then installed his virtual server and accounting application. When the time came to restore the data from the tapes, it turned out they were empty -- they were like new, untouched.
Unfortunately, Greg had labeled all his tapes from Monday to Friday and set up his backup software to write only the tape for the current day. Everyday, he would put the tape in labeled for that day of the week: Monday for Monday, Tuesday for Tuesday, and so on. The problem was, his backup was running at midnight, which was the next day. For almost a year, none of his backups were successful, and he didn't check his logs. Good admin.
As it turned out, his boss and the accounting team had worked extra hours the week before, cleaning all the financial data and closing the fiscal year. All the previous week's hard work was lost -- and so was the rest of the year's.
I don't think he works in IT anymore.
I guess the takeaway leads us back to IT 101: Never, ever take anything for granted -- always work with a safety net. When computers need to get reinstalled at work, I always ask technicians to perform full image backups of the system just in case, even if the users assure you that they have all the data. I can't tell you how many times a user has rushed to us in a panic after we started working on their machine, asking if it's too late. They are so happy when you tell them, "we did a backup anyway, we have all your data."
Related articles
True IT confessions
Supergeeks fess up to some of the dumbest things they've ever done -- and the lessons they learned as a result
Stupid user tricks 3: IT admin follies
IT heroes toil away unsung in miserable conditions -- unsung, that is, until they make a colossally stupid mistake
More stupider user tricks: IT horror stories redux
Idiot-proof your enterprise with these 10 hard-luck lessons of boneheaded IT miscues
Stupid QA tricks: Colossal testing oversights
The trick to nipping IT miscues is testing, testing, testing, as these hard-luck lessons in boneheaded quality assurance attest
Programming IQ test
Are you a script kiddie or a hacker hero? Take our quiz to find out
Even dirtier IT jobs: The muck stops here
More dirty tech deeds, done dirt cheap
The 7 dirtiest jobs in IT
Somebody's got to do them -- and hopefully that somebody isn't you