This is from Quora. My answer is below…
When is machine learning better than crowdsourcin
While it depends on the complexity of the task and the desired quality of the results, if you need to guarantee high quality results, it is best to augment ML with crowdsourcing.
In fact, over the next few years, I think you can expect to see an increasingly symbiotic relationship between crowdsourcing and machine learning.
- Crowdsourcing is an infant industry, the inner dynamics of which are poorly understood and for which there is only the scantest academic research. The innovation that's happening at companies like Crowdflower (disclaimer: the CEO is an advisor to my company) and SpeakerText (disclaimer: this is my company) is just barely scratching the surface of what can be done by applying ML and advanced statistics to crowdsourcing processes. Expect this space to heat up in a massive way over the next five years.
- Crowdsourcing is the most cost efficient way to get labeled training data, and for certain kinds of ML problems––most notably speech-to-text––the size, quality and relevance of your training data set matters more than the algorithm itself. Historically, speech companies budgeted millions of dollars just to acquire a sufficient set of training data. Now, because of the internet and things like Mechanical Turk, you can easily tap into cheap, on demand labor and even turn what was once considered "work" into a game played by people across the globe.
- ML + Crowdsourcing = Supervised Learning. Quite simply, if you first attempt to solve a problem with an ML algorithm––like say speech-to-text––and then assign a human to correct any errors, you can––if you do it right––not only reap huge short term efficiency gains (vs pure human effort), but also improve machine accuracy over the long term by adding more and more labeled training data into the system. The result: constantly increasing labor efficiency, lower costs, and smarter machines––coupled with extremely high quality results.
These ideas are fundamental to the technology that we've built behind SpeakerText. If we're wrong, then the company is fucked. If we're right, well, I'll let you figure it out…and we'll see what happens.
That’s the approach Paul Dix and I are taking with http://market.io. We’re building a product recommendation system that uses heavy machine learning coupled with human intelligence… in this case the keen curation only a good merchandiser can provide.
Matt, you should come to the New York Machine Learning Meetup this Thursday. I’m sure Paul would love to have you give a lightning talk on SpeakerText’s approach.
That’s the approach Paul Dix and I are taking with http://market.io. We’re building a product recommendation system that uses heavy machine learning coupled with human intelligence… in this case the keen curation only a good merchandiser can provide.
Matt, you should come to the New York Machine Learning Meetup this Thursday. I’m sure Paul would love to have you give a lightning talk on SpeakerText’s approach.
Matt,
I was thinking about crowdsourcing the other day in the context of recommendation systems, and how it might be hard to judge how the system is doing if there’s a lack of user feedback (user days “this recommendation sucks” and just leaves). I think making this feedback process fun/addictive/rewarding is key. Wikipedia, where you feel like you’re teaching people stuff, and google’s image labeler http://images.google.com/imagelabeler/ , where you play a fun simple game are just a couple of examples. It seems like the more complex the feedback, the greater the barrier to entry, and the more engaging the mode of feedback needs to be. So my (unexperienced) guess is that you either have to make the feedback dead simple to do (and therefore decrease the complexity of the information you get back) or make it more engaging. Is getting complex feedback ever an issue? Or does it simply suffice in most cases to have a binary feedback that most people will probably ignore but enough will actually do for your system to learn?
Best,
Stan
Matt,
I was thinking about crowdsourcing the other day in the context of recommendation systems, and how it might be hard to judge how the system is doing if there’s a lack of user feedback (user days “this recommendation sucks” and just leaves). I think making this feedback process fun/addictive/rewarding is key. Wikipedia, where you feel like you’re teaching people stuff, and google’s image labeler http://images.google.com/imagelabeler/ , where you play a fun simple game are just a couple of examples. It seems like the more complex the feedback, the greater the barrier to entry, and the more engaging the mode of feedback needs to be. So my (unexperienced) guess is that you either have to make the feedback dead simple to do (and therefore decrease the complexity of the information you get back) or make it more engaging. Is getting complex feedback ever an issue? Or does it simply suffice in most cases to have a binary feedback that most people will probably ignore but enough will actually do for your system to learn?
Best,
Stan
What’s the problem, exactly? Can you share? I’d love to hear more..
What’s the problem, exactly? Can you share? I’d love to hear more..
Good for margins indeed!!!
We know Panos well. Met with him many times. And he appears to like our approach. Thanks for the pointer!
Good for margins indeed!!!
We know Panos well. Met with him many times. And he appears to like our approach. Thanks for the pointer!
Matt,
I think you’re right on the money here and I thought about this implicitly but I couldn’t really formulate it into a succinct post like you did.
I’m currently working on an interesting problem where the problem fits very nicely in the machine learning domain and we’re extremely focused on having a high precision vs recall. I think with Crowdflower and the ease of crowd sourcing today, it might be feasible to achieve this without too much unnecessary effort.
Best,
Mahmoud
Matt,
I think you’re right on the money here and I thought about this implicitly but I couldn’t really formulate it into a succinct post like you did.
I’m currently working on an interesting problem where the problem fits very nicely in the machine learning domain and we’re extremely focused on having a high precision vs recall. I think with Crowdflower and the ease of crowd sourcing today, it might be feasible to achieve this without too much unnecessary effort.
Best,
Mahmoud
ML works on training data sets by recognizing patterns in the data (I’m a signal analysis guy of 14+ years so I’m familiar with quite a few approaches).
Crowd sourcing is a form of voting algorithm when discussed with respect to sites like Digg or HackerNews. Wouldn’t crowd sourcing generate measurements for machines to learn from?
I think you’re describing human in the loop ML which is pretty common for many higher functioning activities.
Most important to my comment, the combination is stronger than either method independently. Eventually diminishing returns come from human in the loop at a low level, and the human roll is pushed ever higher up the stack (and the manpower shrinks). That’s good for margins!
ML works on training data sets by recognizing patterns in the data (I’m a signal analysis guy of 14+ years so I’m familiar with quite a few approaches).
Crowd sourcing is a form of voting algorithm when discussed with respect to sites like Digg or HackerNews. Wouldn’t crowd sourcing generate measurements for machines to learn from?
I think you’re describing human in the loop ML which is pretty common for many higher functioning activities.
Most important to my comment, the combination is stronger than either method independently. Eventually diminishing returns come from human in the loop at a low level, and the human roll is pushed ever higher up the stack (and the manpower shrinks). That’s good for margins!
matt: thought-provoking post, as usual.
for an inspiring example of how a `symbiosis’ between ML and crowdsourcing won a real-world challenge, I suspect you’ll enjoy the story of the winners of the 2009 DARPA challenge: http://bit.ly/9S3qER
(also: for an example of someone looking at the hard data surrounding utility and validity of turker data, I encourage you to check out NYU’s own Panos Ipeirotis’s blog ( http://bit.ly/9XDk9v ) )
matt: thought-provoking post, as usual.
for an inspiring example of how a `symbiosis’ between ML and crowdsourcing won a real-world challenge, I suspect you’ll enjoy the story of the winners of the 2009 DARPA challenge: http://bit.ly/9S3qER
(also: for an example of someone looking at the hard data surrounding utility and validity of turker data, I encourage you to check out NYU’s own Panos Ipeirotis’s blog ( http://bit.ly/9XDk9v ) )