I wonder if a five star rating system could secretly count only the 2 and 4 star reviews without telling anyone, sort of filter out anyone passionate enough to try to game things...
I think the human mind is best at that. Even if there are too many reviews for you to completely read, you can sample them, especially the ones others have rated for usefulness, and with some practice recognize the ones that are passionate with sufficient support.
Specificity is one think I really look for. Those who say essentially nothing more than "It's wonderful" or "It sucks" are noise. Those that say, "It's great, but I'm subtracting one star because the control console is a pain because of X, W and Y" are golden ... especially when people add comments on workarounds....
It's kind of funny, in that this "impersonal" mail order outfit very often manages to provide a better shopping experience than brick and mortar stores for those of us willing to buy things sight unseen. Heck, it's not uncommon for me to check Amazon.com reviews before buying something from one of those.
That might work, but I have doubts. Interestingly, it's the opposite approach the the one taken by pre-employment personality tests, in which each question has no partial credit and exactly one right answer, which is either "strongly agree" or "strongly disagree".
I think Barbara Ehrenreich mentioned that in Nickel and Dimed. At Walmart, one of the hiring surveys included some question pitting personal considerations or ethics against loyalty to management, and the only correct answer was absolute loyalty to management. When she debated the point, she was not offered a job.
In college, I got a part-time job for a stretch of some months grading exercises at the local Kumon center. The job consisted primarily of comparing a filled-out exercise booklet to a printed list of answers and marking incorrect answers. (The other part, the good part, was talking to the kids and trying to help them with problems they didn't get.)
My boss was never happy with the speed at which I worked. Eventually she told me I was too smart for the job and should aim higher. In a vaguely similar way, I don't think Wal-mart is looking for their lowest-level employees to make a lot of independent ethical decisions on the job. If you're going to be hiring unskilled labor, you might not want them to exercise a lot of discretion, on the theory that they're at least as likely as not to screw it up.
IMDB (which is a 10-star system) has a 'secret algorithm' that fiddles with the raw data to come up with a computed score. It's been suggested that some of the factors involved include reduced weight for 1 and 10 star ratings (as well as the age of the user's account - votes from new accounts don't get as much weight, etc).
I'd guess that moderation (as measured by 2-4 as opposed to 1 or 5 stars) is a worse predictor of an opinion's relevance than even the most simplistic automated classification of textual or behavioral feedback.
