This image was lost some time after publication, but you can still view it here.

The upside of that suspect tip I got about what caused Tuesday's massive Internet outages? Tons of tipsters have written in to tell me what they saw on the scene at 365 Main, the troubled San Francisco datacenter whose generators failed during what ought to have been, for the expensively-built facility, a humdrum power outage in the city.

The very best report has come from an employee of a Fortune 500 tech company who was inside the datacenter at the time of the outage. He witnessed the chaos and confusion at 365 Main firsthand. Let's just say he was not impressed by how employees at the datacenter performed during the crisis. "You should be able to trust the power," says my informant. "That's what we pay them for." After the jump, his complete account.

FYI, I was in Colo 7 when the outage happened, preparing to kick off a very large upgrade to a very important set of gear in my employer's cage, and so when the power flickered off for a second, I nearly peed my pants. Not only did the power flip off, but there were incredible scary noises from the roof and the HVAC as stuff was turning on and off and generators spooled up and emergency lighting flipped on and off. I ran for the NOC, and got ahold of somebody who didn't know anything who I hung up on. Seconds after that, every door in the place burst open, and 365 Main folks were sprinting down the halls.

Colo 7 is on the top floor, and the only way to the roof seems to be through a stairwell in the big hallway up there outside Colo 7 and 8, so I watched for a bit while a coworker made the call to management. A lot of people went running by to the roof and towards the big PDUs in the center of Colo 7 and into the NOC/telco area, etc., and I'll tell you, I didn't see any drunk 365 Main people anywhere, just folks with an "OMFG, what just happened!!??" look on their face running around. Soon after that, other sysadmins began to trickle in, but we were too busy making sure that our own stuff was coming up to pay much attention. I've never seen so many people in 365 Main before.

Anyways, I saw the employees running around and flipping out and it was very exciting for everybody. Seems to me that if some guy had hit buttons, then a lot of activity would have been centered around the area with the buttons (possibly revolving around doing bodily harm to hypothetical drunk people), not the roof or the NOC or the PDUs or anything else that I saw the brown shirts doing.

So you might want to just kill that rumor. Really, don't you think that 365 Main would LOVE to be able to pin a quadruple failure of their equipment on some alcoholic? If it were a drunk guy doing it, it would make me a lot more confident in 365 Main, because it's easy to fire the guy and require breathalyzers to get into critical infrastructure rooms or whatever. As it is, I wonder what will happen next time we have to go off the grid.