Today concludes what I would absolutely consider the week from hell. It is very fitting that tomorrow is Thanksgiving and we get a short break after the time that I have put in over the past week. In case you missed it, I came into work a week ago to find that our primary domain controller had failed to boot back up after a power outage.
Unfortunately, this server was not only our domain controller, but it also hosted our DHCP server, DNS, and a handful of other services. That pretty much meant if you didn't have a static IP address (which is most people in the organization), then you had no Internet.
Wireless was down because the access points and wireless networks rely on DHCP to get addresses on the network. Phones were down for the same reason as well as hard wired machines. File services was running on a separate piece of hardware, so none of our data was lost.
Virtualization is an amazing thing and it saved my butt in this case.
With the server down and everyone looking to me for a solution, I started attacking the problem from three different directions. I was hoping that at least one of them would net me some results.
The key issue with the server was the fact that although the drives in my RAID array had been healthy two days earlier when I looked a them, after the power outage two of them had failed. With RAID 5 you are usually okay if one fails, but if two fail, then you are hosed.
I threw a couple of spare drives into the array, but that didn't seem to help. My other two options were to start from scratch with a new virtual server or access our backup appliance that had a bare metal backup of the server on it and spin that up in the cloud.
It took forever to coordinate with the organization where our backup appliance is hosted, and once they did get us set up, I was able to boot the server into the cloud, but I had no connectivity to it.
So really my only option at this point was to start from scratch which I had been working on between all of the other things.
Creating a new server is pretty easy. Like I said, virtualization is amazing. Within half an hour I had a fully running Windows 2022 server ready to go on one of my virtual hosts. I was able to use my backups and pull our DHCP configuration from the old server and apply it to the new one.
That got us 80% back up and running.
With the primary domain controller down though, people still weren't able to authenticate to the network and access their files. Once again, in a stroke of luck, we had our active directory replicated down at the same place we host our backups. After a considerable amount of time, they were able to promote their replica to the master and then I added my new server as a replica.
Long story short, we got back up and running. It took all day though and looking back, if I could have cut out all the waiting on the other parties to do what I needed them to do, I think I could have had it fixed in two to three hours.
After a week from hell, we are finally back up and running 100%. I was able to clean up the remaining services that were broken over the weekend and earlier this week. Moving forward, I am going to have my own replica on site running on separate hardware. I don't want to have to rely on anyone else if this happens again.
I also stored a copy of our DHCP configuration in a safe place so I can easily access it in the future should I need to spin up a new server on the fly.
Plus, I get to buy a new server, so that is pretty cool. I had been talking to my business manager about a new server anyway. She was hesitant, but after this happened, she basically said "buy whatever you need". I have a shiny new Dell PowerEdge server being custom built as we speak.
I am super excited about getting it in place and starting to set up our in house redundancies.
After a week like this, I take peace in the fact that tomorrow is Thanksgiving. Weeks like this past one are honestly few and far between in my job. It's relatively stress-less and when things are working they just work.
I think weeks like this past one remind us why all the other ones are just so special.
Sports Talk Social - @bozz.sports
