Indeed, Panos's remarks are spot-on here. Reputation and quality management on small amounts of work are really only the smallest problem in crowd computing, and have been artificially created by the lack of simple reputation systems in broken markets like Turk.
This is one reason why next-generation crowd platforms like our own, MobileWorks, (and for that matter, why users of crowd platforms) should make use of a high-quality workforce as a starting point, rather than as their ultimate goal.
Once you assume that you have a high-quality workforce, a host of other problems become more relevant and valuable.
- How can we make it simple for a user to specify their tasks in a way that crowd workers can understand?
- How can we use crowds to solve arbitrary tasks, rather than a limited range of tasks?
- How can we give back responses in real time?
One of the reasons that MobileWorks (www.mobileworks.com) has been so effective as an alternative to Mechanical Turk is that it goes beyond reputation to address these problems.
here's my take, for the tasks I've done with turks. Maybe I'm no expert, but I've run enough HITs to get an account rep.
A certain group of turks are just criminals who do totally bogus work. Maybe they use automation or maybe they just click as fast they can by hand, but their work is worse than an optimal model that does work by chance. They're easy to catch, both technically and emotionally.
The people I have a hard time with are the "superturks" who do a lot of work and manage to do it just barely well enough that it's hard to kick them out. Technically it's easy to catch them because they are the N people who did the most work. Maybe I could just be tough and kick them out, but I'd get the work done more slowly without them.
My guess is that a turk who's good at task A might not be good as task B and vice versa. I know that turks like my tasks enough that I get regular emails begging me for more HITs...
... but then, I hate to break the news that I built an A.I. that's more accurate than the average Turk and 1000 times cheaper.
This is one reason why next-generation crowd platforms like our own, MobileWorks, (and for that matter, why users of crowd platforms) should make use of a high-quality workforce as a starting point, rather than as their ultimate goal.
Once you assume that you have a high-quality workforce, a host of other problems become more relevant and valuable.
- How can we make it simple for a user to specify their tasks in a way that crowd workers can understand? - How can we use crowds to solve arbitrary tasks, rather than a limited range of tasks? - How can we give back responses in real time?
One of the reasons that MobileWorks (www.mobileworks.com) has been so effective as an alternative to Mechanical Turk is that it goes beyond reputation to address these problems.