The unthinkable had happened, and I found myself in the middle of a catastrophe. The server had crashed and my precious backups were useless. It seemed that if not for a stroke of luck, all the data would have been lost. But that still left months of data missing…
Mom Always Said To Dress In Layers
The reason I do manual backups is that I just have a gut reaction that if I don't see the job done, it might not be. I've worked with electronics in one form or the other since the late 70s, and I don't trust the stuff. Granted, reliability has improved over the years, but still, old habits are hard to break. I'd rather do it myself than trust a machine to get it done for me.
But as it turns out, I do ask the machine to back up this program for me--every day. You see, even though I don't trust them, I do use them, just in case I am the one screwing up. Which, in this case, for whatever reason, I was.
Redundancy is the guardian angel of IT. When it comes to my work I believe one thing: eventually everything's going to fail. No matter what I do, I'm gonna miss something. I do very good, very precise work, at breakneck speed, deliberately and meticulously. I know that I am working a bit faster than normal, and I know that means I am statistically more prone to make mistakes(maybe). So instead of slowing down too much, I always add little safeguards against my own blundering as I go.
It's the layers principle. Try to get as many layers of security between you and whatever it is that you need to be protected from. In this case, loss of data. Once I recovered from my initial panic and remembered the automatic backups, I was able to restore the data files right up to the night of the crash. It would be no exaggeration at all to say it was an emotional moment to see the program working as if nothing had ever happened.
She Also Said There'd Be Days Like These
I was on top of the world, almost giddy. The program was happy, I was happy, my customer was happy--there were probably cartoon birds flying around somewhere singing just for us. Triumphant, I returned to the server room ready to install the last program, restore its backups and still get done in time to eat supper. I placed the disk in the tray and gently tapped it in. There was no documentation with this cd. I had been told that the company which released it hadn't been heard from in about 5 years. But it had always been a straightforward program, so I wasn't worried in the least.
There was no autorun on this cd, so I told the machine to display the contents. There was one file, an executable installer, so I ran it. It wasn't a large installer, only about 20 megabytes, but it seemed to be taking a very long time to install. It finally showed the Finished screen, and I clicked Ok. The backup file for that program was a single database file, so I grabbed it from the backup drive and copied it into the folder. This file was pretty large, about 22 meg, so I sat back to wait the 3 or 4 minutes I figured it would take.
A couple of minutes went by. I noticed that the "Time Remaining" count wasn't changing. Well, sometimes it doesn't. I gave it a few more minutes, then hit the keys to bring up the task manager. And... nothing. The machine was locked up in some loop somewhere and I wasn't invited. So I held the power switch in for 4 seconds and restarted the machine. Sometimes these things happen, I've seen it a hundred times. No problem, just do it again.
Only, this time... well, I don't know if it was my mom, but somebody said there'd be days like these.
The Crash... Again?
Yep, again. On the reboot, Windows wanted to scan the disk, and I thought that was a good idea, seeing as how it had just locked up. So it scanned, found and repaired the offending area of the disk, and rebooted. As the desktop was coming up, it detoured and decided I wanted to see that beautiful Blue Screen of Death again. I tried restarting several ways, but each time Windows would get almost to the desktop and then shut down with an error code that could mean lots of things.
First I tried repairing the master boot record, but that wasn't it. Then I tried repairing the Windows installation itself. No help there either. After a couple of hours I was still getting blue screened on every startup. There were several things I could have tried at that point. Since it wasn't repaired by repairing Windows, I could look for it, maybe get lucky and find it in another hour or so. More likely was that I would spend many hours and never find it.
So I decided to just reinstall Windows, replacing the existing installation. It would mean that I had to reinstall all the updates and programs again, but that might actually be faster than trying to find the source of this latest crash. So far, the hardware seemed to be working just fine. All the installing and repairing indicated that the components were in good shape. So it was almost certainly software.
Never Underestimate The Value Of Coffee
I did a replacement install of Windows, and soon was back up on the Windows Update site grabbing the 94 updates I knew I would need. A few hours later and I was back to installing that monster management program. The install was going great, until I restored the backed up files. Sometimes, when dealing with databases, after restoring backups, a little routine has to be run that rebuilds the database structure, optimizing your files and your settings. Every time I tried rebuilding the database, I would get a multitude of errors from the program about missing files.
From the pinnacle of victory I found myself cast into the depths of despair once again. I'd spent too many hours not to see any results. Outside, it was dark. It was cold, the January wind moaning low between the buildings and the trees. I was tired. It was late. I was hungry. I wanted a cup of coffee. Yeah, a cup of coffee and about 7 hours of sleep. I knew it was time for decisive action. I shut it all down and left the building, heading for food, a good cup of coffee, and a night of fitful sleep.
Clickback again tomorrow, for the continuing saga of... Catastrophe!
.
See Part 1 HERE
