Two studies explore the murky morality of online reviews
Two studies explore the murky morality of online reviews
Oh, what a tangled World Wide Web we weave.
Take "deceptive opinion spam," a virulent new strain of flimflammery that's a pox on consumers everywhere. "This is not the spam we've become accustomed to—those annoying, unsolicited messages that pop up in e-mail," says communication professor Jeffrey Hancock. "Deceptive opinion spam consists of fictitious opinions that are deliberately written to sound authentic. That takes deceit to a whole other level."
Anyone who shops online is familiar with the star rating system and accompanying testimonials written by customers who have allegedly purchased or sampled the goods. Millions of these reviews are honest. But deceptive opinion spammers have infiltrated this trust-based system and, for as little as five bucks a pop, lavish praise on products and services based solely on their imaginations.
Consider this five-star review of an Ithaca dental practice: "The best dentist ever. Always in a cheerful mood, never down. Just all around wonderful people." Sounds great. Until you research this particular patient's past reviews and find that he also gave five stars to a roofing company in Tulsa, the Bedford Plaza Hotel outside Boston, the Brazilian Grille in Boise, and another hotel in South Dakota. The reviewer's home base is San Diego. All seventy-four of his reviews are gushing.
You don't have to be Miss Marple to crack such a simple case of deceptive opinion spam—but it's surprising how easily fooled we are by online hokum.
According to Hancock, that's normal. Research on deception shows that we are hard-wired to believe. It's called the truth bias, and it makes civilized society possible. Violating that basic trust at the point of sale has untold economic impact. The opinion spam contagion is believed to be widespread, infecting all sites that use customer reviews, say Hancock and his colleagues; juicy targets include hotel, restaurant, and travel sites like Tripadvisor, Citysearch, and Yelp, as well as general retailers like Amazon. But no one knows how prevalent it really is, and people are not good judges of online writing. "Humans have had about 60,000 years of face-to-face conversation, so we've identified cues for spotting lies," Hancock says. "But we've only been communicating in virtual ways for a relatively short time. We're not very adept at identifying online deception in text."
While the technology for detecting digital deception is in its infancy, Hancock and computer science professor Claire Cardie, along with grad students Myle Ott, MEng '07, and Yejin Choi, MS '09, have developed software that exposes deceptive opinion spam. By integrating principles of psychology and communication with linguistics and computer science, the researchers produced an algorithm that scans text for patterns of prevarication based on a first-of-its-kind collection of opinion spam.
Four hundred fake reviews written by known hired guns and 400 (presumably) honest reviews of Chicago hotels were fed into the system. In repeated tests, the software weeded out the frauds with about 90 percent accuracy. (By comparison, a control group of three Cornell students fell around the 50 percent range, about as good as guesswork). The program detects fraud based on parts of speech and other tip-offs: deceivers tend to use the first person singular "I" and "me" as if to validate their observations; they often mention traveling companions and their reasons for being in Chicago (evidence of "overcompensation," says Cardie). And, like fiction writers, deceivers deploy far more verbs, pronouns, adverbs, and superlative adjectives.
Truthful reviews, on the other hand, describe concrete spatial details like bathroom size. Also, they use more punctuation and often employ keyboard short cuts like "$" as well as a liberal spray of dashes, ellipses, and parentheses. Honest folks stick with meat-and-potatoes nouns, prepositions, and adjectives and eschew the exclamation point.
Cardie's team is well aware that publishing these tricks of the trade will inevitably generate more deceitful prose. But the more that businesses use spam to improve their bottom line, the greater the risk of having their credibility sabotaged by hired hacks and insiders seeding their own glowing reviews. Says Ott: "If there is an overabundance of positive reviews, the whole system of consumer reviews as a sales tool falls apart."
And while the algorithm has its limits (so far, it is valid only for Chicago hotels), the software could be adapted to flag suspicious reviews; companies could then block writers with a known history of lying. The next step is to try the approach in other domains such as restaurant and book reviews. The industry response to Hancock and Cardie's study was overwhelming: The researchers were besieged by job offers and inquiries from Google, Amazon, Tripadvisor, and various hotel chains.
Opinion spam comes in a variety of flavors, some less overtly destructive than others. For example, there is the kind that some Amazon reviewers generate simply to boost their own egos; writers whose reviews are deemed most helpful can generate a following. A study profiling Amazon's top 1,000 reviewers, led by Cornell professor Trevor Pinch, reveals a world of bored freelancers, retirees, and armchair experts vying for most-helpful status by cranking out hundreds of reviews of items as disparate as romance novels, exercise machines, and dog brushes.
Pinch's study provides a composite of the opinion-spammer-at-large: roughly 70 percent are male and 40 percent are professional writers. Eleven percent are retirees, a fact that amuses him. "After my dad retired, he used to enter all these silly word game competitions and win gifts," muses Pinch, a professor of science and technology studies and of sociology. "It was a nice little hobby for him." But there's a darker side: Pinch found that 85 percent of top Amazon reviewers got free products from publishers, agents, authors, and manufacturers. Another 78 percent said they often or always review such giveaways. "If you get something for free," says Pinch, "a customer review is really not a customer review anymore."
Until technology catches up with the digital deceivers, a traveler's advisory for online shoppers is in effect: if something sounds too good to be true, it probably is. As the old algorithm goes: Caveat emptor.
— Franklin Crawford