This image was lost some time after publication, but you can still view it here.

I'm a dunce. I was wrong. There, I said it. In running a tip on Tuesday that a drunk employee brought down 365 Main, the San Francisco datacenter which hosts servers running some of the Web's most important sites, I trusted a source I shouldn't have. Here's the story behind my 365 Main post. A warning to readers of sensitive dispositions — I'm about to take you inside the sausage factory, and it's a bloody mess.

The tip, which I printed verbatim, came in by AOL Instant Messenger — not the most trustworthy of mediums, I'll grant you, but one favored by sources trying to preserve their anonymity. Despite my misgivings, I trusted the tipster for the following reasons:

  • He IM'd me at 2:14 p.m. Pacific, only 27 minutes after 365 Main lost power supply from PG&E, before anyone outside the datacenter knew details of the outage, claiming to have information from a friend who worked there.
  • I pressed him for some details and established that his connection to 365 Main was plausible.
  • He gave me a first and last name, and told me that he worked at a large software company. I established that there was a real person with that name who worked at such a company.

Contrary to what some blogtards have written, I didn't invent the tip, or make the tipster up. We had sustained IM and phone conversations. And my tipster, for what it's worth, continues to stand by his version of events. But I'm the ultimate blogtard for not checking him out more thoroughly. I've since learned that the tipster has associations with a band of hackers who delight in social engineering, the art of using technical means to get human beings to do their bidding. And if he belongs to such a band — which he denies — I played right into their hands.

Why? I'm told by people close to the hacker group that they do this kind of thing for sheer amusement. That they enjoy harassing people in the Web 2.0 world. That they have broken California's privacy laws by illegally recording telephone conversations, among other misdeeds. (I'm not going to name them, because they'd just enjoy the attention.)

Before you condemn me, though, let me say this: Yes, I'm an experienced reporter who ought to have known better. While I worded the post itself cautiously, the headline should have reflected that same caution. It didn't, and I apologize as well for that. But in running Valleywag, I'm experimenting in a new medium and a new style of journalism. Yes, I take tips over the transom. Yes, I post early and often, and factcheck, update, and correct as I go. And yes, this is why you read blogs.

And I also should say a bit about why I was inclined to believe the tip. 365 Main CEO Chris Dolan personally gave me a tour of his facilities a few years ago. He showed me the generators, the flywheels, the systems. He led me to believe that 365 Main could ride out an earthquake, with its supplies of fuel and water.

Based on what Dolan told me, I found it more plausible that an employee, acting maliciously, could take down 365 Main's power than something as insignificant as a power outage. Protection from power outages, after all, are precisely why companies put their servers in datacenters with supposedly redundant systems.

I hope no one will come away thinking that, because of my error, 365 Main should come away from this free from blame. Big questions remain about its outage. The company itself admits it still does not know why half of its generators failed. Four of those five generators failed because of "problems in [their] start sequence."

Think about that: Why would four generators fail, for the exact same reason, at the same time? I have reason to believe that, for reasons of his own, my tipster may have been trying to deceive me, and through me, you. But I ran with the tip because it had the ring of truth. Something very wrong happened at 365 Main on Tuesday, and we still don't know what it was.

365 Main has denied that there was any "employee misconduct." With its investigation admittedly incomplete, however, I'm surprised it would make such a claim. As I've pointed out, the datacenter's credibility is very low. (And before you say it, allow me: As is mine.) Customers are fleeing fast. My latest tip, if you care to believe it: Technorati plans to move 500 servers out of 365 Main by September.

There's a simple step 365 Main's management can take to staunch its customer losses: After finishing the investigation, post the video from the datacenter's 24-hour surveillance tapes to the Web. Show exactly how its employees behaved during the outage. Or they could release the tapes to me, and I'll review them, and post about what I saw.

Feel free to doubt me. After relying on a source I no longer trust, I deserve it. But until management at 365 Main concludes its investigation and releases its surveillance videos to back up their version of events, you should continue to doubt them, too.