My job is pretty stressful. I have to maintain over 500 hospital servers. I access all of them with root access. To the non techies, this is basically God mode.
In my department, almost everyone has at least one "Whoops!" moment where a second of bad judgment brought an entire hospital to screeching to a halt and lost 100s of patient records, because the site had to restore from backup.
I pride myself on having never caused one of those. But, due to my profane use of language, and my the fact that I think in profanity, my documentation sometimes throws people off guard. This morning, I got a call at 3:00am that a server crashed, and the hospital is in bad shape. This hospital is operates completely electronicly. The backup server failed, too.
I work with a guy over the phone for 3 hours to get the server up to a point where it can limp along and get me vpn access in so I can work on it. I fixed the issue, and saved the day.
Since this is a fairly severe situation, I have to document everything, and send mail to my manager, director, CIO, CFO, and both VPs.
This should have been my moment to shine. But, I instead I sent company wide mail to this effect:
"Three dick (disk) drives failed on the user raid 5 array. I was able to boot the server from the boot disk, and remount slash. I pulled the OS from this morning's backup tape and was able to move it in line. Some data was corrupted, but I was able to rebuild AR database. Only issues remaining are pharmacy interfaces are down, and the nurses are having problems printing faceshits (face sheets)"
The running gag at work is people get that nickname based on the first thing they fuck up. This is one of the most embarrassing things I have ever done. On Monday, I just know my new nickname is going to be dick drive or faceshit.
So, please tell me your stories, and please tell me one that is worse than mine so I can feel better.
