Pogue agrees — advance gadget reviews are bogus
New York Times gadget reviewer David Pogue got into an email back-and-forth with Valleywag after he was tricked into writing an article by advance misinformation on a pre-launch product. In theory, it's good for reviewers to test and write up products before release day, so consumers can make informed choices. In practice, Pogue and we wish the industry standard would change.
The problem with tests based on advance access to carefully doled out "review units" is obvious: It's in the gadget maker or service provider's interest to provide a few well-placed reviewers like Pogue with hand-picked, flawless gear and overly helpful support. The real customer's storebought or online experience is sure to be different.
For example, my own Macs have a lot more hardware failures than Walt Mossberg's do at the Wall Street Journal lab. And I've frequently been given misinformation from the Genius at the Apple Store counter. He can't grab a senior product manager to answer my questions, as the company's publicists do when I'm working on a Wired review.
In Pogue's case, he reviewed an overseas phone calling service whose super-low rates — the feature that sold him — turned out to be three, four or more times higher when the service went live.
Pogue observed to us that the Times' restaurant and theater critics do their reviews incognito, buying their own tickets and meals on the company expense account to avoid special treatment or pricing. Yet gadget reviews are traditionally done — by the Times and everyone else — on equipment loaned by the maker. Reviewers are under pressure to seek early access so they can meet deadlines and beat other publications, rather than wait for the product to hit stores or the site to go live.
The shining exception is nonprofit Consumer Reports. All tested products are purchased at retail by its staff, and no free samples are accepted from manufacturers. But who's going to wait around for CR to pronounce that "Our tests of Apple's new iPod Touch confirm that it is indeed essentially an iPhone without the phone?" By the time early buyers learn, as CR did, that the iPod Touch's screen can be dimmer than an iPhone's under the same lighting conditions, it's too late.
I don't expect David Pogue to stop accepting advance products and services, nor the special treatment that comes with them. It would be career suicide unless all other journalists stopped at the same time. But when readers' experiences turn out different than that provided to reviewers, hopefully more writers will do what Pogue did: Stand tall and warn everyone, rather than letting the issue die quietly on a Corrections page.
