12:12pm rolled around and I had plans to meet my friend Kevin Mitnick for lunch at one of our favorite lunch spots at 12:15pm. I hopped into my car and started heading to lunch. At 12:15pm I received a text message from Rain Man -- "Error Establishing Database Connection." I called Rain Man's cell and we had a brief conversation to determine that I did indeed need to abort lunch. On my way to the office I called Mitnick and let him know I wasn't going to make it to lunch. This was fine because Kevin has a tendency to do the same thing on a disturbingly regular basis. Rain Man and I actually refer to the act of making plans then cancelling them right before or shortly after they are supposed to occur as "Mitnicking out." I got to the office and determined that once again our VM storage pool was full. For some reason I was calm. I began moving more data off of the NAS and made the decision to call 1-800-KIL-LMEE. I navigated through the phone menus and a man with a thick accent answered the phone. I'm sure he wanted to help, but he mumbled and I was stressed. After asking him to repeat almost every question for clarity, I was eventually escalated to the next tier of support. Because I already know how this story goes, let's call tier 2 support guy Dumb Ass. Dumb Ass asks me some questions and thinks he's on his way to knowing what is going on with our system. He sends me a KB article referring to a bug in the de-dup software that will leave residual meta-data following a de-dup process. Over time this data builds up and fills up a disk pool. This explains why I can run "du -h ." on the file system and see that I'm only using 1.8TB while the storage appliance thinks I'm using 4TB. Dumb Ass has me type the last command to get the information that confirms this perfectly sound theory. The command did not return the output he expected at all. Something was wrong. He had me type a few more commands on the appliance's shell with a special privilege mode that only gets used when you're totally fucked. Nothing.
After 45 minutes or so of working with Dumb Ass and explaining that this was a mission critical machine that affects 150 people when it goes down at any point in the day, I was escalated to tier 3. In my experience tier 3 support is where you talk to somebody who really knows what's going on. These guys are the guys that have a deep understanding on how a system works and they are generally much more aware of all the tricks for their company's systems. I value tier 3 support and the poor souls who work at this level. They deal with real problems every day of their lives. They probably fund the R&D departments at all the major pharmaceutical companies. Anyways... Stan the Computer Man and I had a brief overview of the problem and got to work. We ran a few more tests and Stan thought for sure this was probably the bug Dumb Ass thought it was. But why did the diagnostic command return a result that was something other than useful. We all found this troubling. Stan then asked me "what commands did you type with Dumb Ass?" I walked him through the commands we had executed then I told him that we hopped into the magic oh shit privilege level and executed the oh shit command. Stan was not happy. I think I heard Stan's head explode. He confirmed "so you typed the oh shit command?" I replied with a simple "yes." In concert we both sighed "oh shit." Stan explained "you see, by executing the oh shit command you blow away all of the useful logs one would need to debug a problem that isn't documented. Because the diagnostic command is returning odd results, I could really use that data so that we can look into what's going on with your system." We were up shit creek. Stan put me on hold while he talked to some colleagues to see if they had any ideas on how to move forward. We all had the same simple plan that is less than desirable... I was going to have to wait for the bug to pop up again.
It was 5:09pm and I'd been watching the system waiting for the free space to start going down. Instead, all it did the rest of the day was go up. Next thing I knew, there were 3.52TB free on my NAS. WTF!
Let's rewind a bit. I had a ticket to an award ceremony hosted by Young Tool Bags of The City that Scrooge Mcuck had asked me to attend so that our organization had some representation. We were also up for an award. I asked what the dress for this event was and Mcuck told me just to go in whatever I wear to work, I knew he was wrong. I usually dress in denim jeans and a polo shirt. Today I wore a nice button up shirt with slacks and dress shoes. I couldn't find my belt, so I skipped that. I briefly considered wearing my jacket and a tie, but the way Mcuck was talking, this was a pretty casual event. After all, it was hosted by Young Tool Bags of The City and those guys aren't exactly the definition of classy. On top of the little event Mcuck was sending me to, Spools, my tape vendor, asked me if I'd like to grab a drink at 8pm. Spools is a fun guy and has a lot of the same tastes for bars as I do, so I didn't want to turn him down. It was just going to have to be a long day.
At 5:30pm I walked from our office to the country dance hall where the event was being held. I want to repeat that. A business award ceremony organized by Young Tool Bags of The City was being held in a country dance hall. I know I reserved the right to creative freedom at the beginning of this fine publication, but I can't possibly make that shit up... Anyway, I got to the dance hall and there were cars lining up at the free valet. I walked into the entrance, they checked my name off the list, and I got my hand stamped. I walked in and the first thing I saw in the country dance hall was a woman suspended in the air doing aerobatics. She wasn't very good, but that's irrelevant because it was totally the wrong venue for that kind of art display. I b-lined for one of the 4 bars that were opened and asked the unhappy looking bartender how drinks were working. She informed me that it was a pay bar and that Young Tool Bags of The City weren't buying any of my drinks. I contemplated asking her what the fuck I was there for, but instead asked for a water. She just stared at me as if water was something she'd never heard of before. I got my wallet out, put a dollar in her jar and suddenly water was being poured into a flimsy plastic cup. Man, Young Tool Bags of The City really over did themselves with this event. With my water in hand, I was ready to scan the room. I took a quick inventory of all the folks there to figure out where some other sad IT people were. There were none. AND I was under-dressed. Everybody was in full business or cocktail attire. With no other socially awkward IT people, no other under-dressed people, a girl hanging from the ceiling of a country dance hall, and no alcohol to act as a social lubricant, I was becoming very uncomfortable with the situation I'd put myself in.
The food arrangement was also strange. People had paid serious money to be at this event and there was no real entree like food. It was buffet style with tiny plates for things like bruschetta, pasta, and salad. I guess I was basically looking at heavy horderves with an absent cheese platter. Whatever. I filled my first plate with humus, some artichoke dip, and a little caesar salad. I found a high top table away from the crowds and began eating my food. Eventually I heard Spit Fire's alto voice ring out over the crowd "Brady!" She introduced me to her correctly dressed husband and we all talked for a few minutes before they ran off to find some food. I guess Merlin saw us all and eventually joined the table. Spit Fire and Merlin asked about the problem I was working on at the office and asked how it was going. I explained that free space was disappearing into thin air and that it wasn't fixed yet because we had to let it happen one more time before we could figure out what was wrong. That conversation reminded me that I needed to check in on things. I pulled out my phone and sshed to the supercomputer. It was now 7:00pm and we were up to 3.76TB of free space. I was baffled. I used this situation as an opportunity to Mitnik out. Once back to the office, I stared at my monitors in fear. When was disaster to strike next? Why was everything working so well?
My phone vibrated to remind me of my drink with Spools. I walked to his favorite bar in town and he asked if I'd figured out how the disappearing act worked; I guess Rain Man had told him about it. Spools being the nice guy and/or very good salesman said "I know Spools probably isn't of any use to you in this matter, but you let Spools know if there's something he can do and Spools will do it." I told him to buy me a drink. He grinned and so began a night on the town with Spools.
I had a Manhattan with my favorite rye while he had something the bartender described as a "thoughtful tasting drink." Eventually Rain Man showed up with his boyfriend and a house guest from out of town who was visiting for the epic party Rain Man's boyfriend throws every year. I ordered water as my next round while Rain Man and company caught up. We all decided to do a second round, so I had a second Manhattan. I had a nice buzz going and had loosened up a bit. I was less worried about the eminent melt down at my office just down the street, but I was still checking my phone every minute. We finished our drinks and Spools was ready to go to another bar. Rain Man and company were ready to get drunk on somebody else's dime, so they were enthusiastic about Spools' plan. Rain Man being the cool boss that he is told me not to go back to the office and to join. Hell yeah!
The second bar of choice for Spools is right next to my condo and down the street from his hotel. Spools road to the bar with Rain Man's man in his fun but terrifying 1960's VW Bug converted into a buggy. Rain Man and I walked back to the office's garage and I updated him on the shitty Young Tool Bag shindig and the storage situation. He was disturbingly cool with the fact that we were basically waiting for a time-bomb to explode in our datacenter. Though, Rain Man doesn't really worry about much of anything. At bar #2, we had a couple more drinks and called it a night around mid-night.
Overall, this day was a weird one.
