Over the weekend, the Bookstore lost their A/C unit and burned their servers to a crisp. I got called in on Monday, one of the machines had it's 2 mirrored drives with it's OS on them none functional. I was able to get one of the disks to start working after getting them cooled off.
Since that left one failed disk, I worked from the bookstore today. I was doing pretty good, I got HP to get a drive down to us, got the mirror synced back up, and basically had them completely back and stable by 1pm. (Where it looked very bad on Monday at the same time.) We talked about, and decided to patch the server while it was down. I had patched their stage, no problem, but after patching this box, it didn't boot up again. The box had a special piece of HW installed I think the patching problem was related. (It's complicated, and I don't want to really get into it right now.)
Basically, I rescued their machine, got it all back up, then completely toasted it. Since the bookstore locks up at 6pm, I took their disks to work on them on my machines in the Data Center. I took a 4 hour break for Stake Interviews.
Now, 14 hours after toasting the machine, I think I have it back to where it was. It took a great effort of skill and a little luck, but this is the last time I am going to say it, I don't want to curse myself for later when I go to try to plug the disk into the machine with the special piece of hardware... (And people tell me I should be a little humble, at least.)
No more posts until something non-work related occurs. I'm hoping that's soon.
No comments:
Post a Comment