Archive for the ‘CMU’ tag

Can crowd work (or citizen science) become a career option?

By February 7th, 2013 at 10:24 am | Comment

From Carnegie Mellon University

Research could ensure that crowd work becomes a career option, not a dead end Carnegie Mellon scientists and other crowd work researchers issue call to action.

PITTSBURGH—Crowdsourcing is an effective way to mobilize people to accomplish tasks on a global scale, but some researchers fear that crowd work for pay could easily become the high-tech equivalent of a sweat shop. Trivial work for rock bottom pay isn’t inevitable, however, and they’ve outlined a research agenda to make crowd work both intellectually and monetarily rewarding.

Leading researchers in crowd work from Carnegie Mellon University and other institutions will present their plan, hashed out in a special workshop last spring, at the Association for Computing Machinery’s Conference on Computer Supported Cooperative Work and Social Computing, CSCW 2013, Feb. 27 in San Antonio, Texas.

Finding ways to enhance collaboration, incorporate artificial intelligence and create ways for workers to build reputations are among the research challenges ahead.

“When my baby daughter was born I asked myself, ‘would I be proud to see her grow up to be a crowd worker?'” said Aniket Kittur, assistant professor in Carnegie Mellon’s Human-Computer Interaction Institute.

Co-authors with Kittur of the research strategy include Jeffrey Nickerson, director of the Center for Decision Technologies at Stevens Institute of Technology, and Michael Bernstein, assistant professor of computer science at Stanford University. Other leading crowd work researchers from Carnegie Mellon, Northwestern University and the University of Texas, Austin also contributed to the report, which is available for download.

The crowd work industry has expanded rapidly in recent years, with a number of vendors now offering work for people who get paid per task or who compete for prizes. A prominent vendor is Amazon Mechanical Turk, which claims more than 500,000 workers in more than 190 countries who complete tasks that, in some cases, may take only seconds to perform. Another, CrowdFlower, says it can access more than 2 million contributors worldwide. Others, such as oDesk, provide skilled labor, including web developers, designers and translators. Platforms such as Innocentive invite people to invent solutions to problems in hopes of winning a prize.

The still-young industry could grow very large because portions of almost any job — perhaps as much as 20 percent — potentially can be sent “down the wire,” Kittur said. Many crowd workers are paid substantially less than U.S. minimum wage, however, and, left to market forces, the crowd workforce could remain stigmatized and exploited.

“What if I want access to the best people in the world, but for only five minutes of their time?” Kittur said. As beneficial as that might be for some businesses, that possibility will not be achieved if the crowd workplace isn’t attractive for the very best workers and thinkers, he added. The call for action by Kittur and his colleagues, also discussed in a recent post on the Follow the Crowd blog, envisions three major research steps:

Create career ladders. Research is needed in how to structure teams so that skilled workers can train novices, as well as help design jobs and catch problems. Mechanisms are needed for credentialing workers. A better understanding of worker motivations could lead to better job designs.

Improve task design through better communication. Research suggests that some quality problems in crowd work have more to do with poorly designed tasks than with unskilled workers. Artificial intelligence could be used in complex tasks to identify work products that might still need improvement and assign workers accordingly. The crowd itself also may be used to train the computer programs, helping them support a broader range of tasks. Improved instructions and feedback mechanisms likewise could improve the work product.

Enable learning. Quality assurance assessments can identify skills that workers need to polish or learn to tackle new work tasks. Online tutoring, combined with tracking of work history, could support personalized instruction and feedback. The work platforms themselves will need mechanisms for learning what kinds of work requests attract talented workers, recognizing the patterns of learning and skill building among workers and determining what tasks are appropriate for which types of workers.


This work was supported by the National Science Foundation, a DARPA Young Faculty Award, a Temple Fellowship, Northwestern University and the Center for the Future of Work in Carnegie Mellon’s Heinz College.

Follow the School of Computer Science on Twitter @SCSatCMU.

Contact: Byron Spice

Categories: Citizen Science


EteRNA: Biology plus videogames equals cutting-edge science

By January 14th, 2011 at 7:35 pm | Comments (6)

Learn how RNA folds, then create your own shapes to test in the lab. Photo: EteRNA

Learn how RNA folds, then create your own shapes to test in the lab. Photo: EteRNA

What class of molecules dominated the primordial stages of evolution, and seems to function as an exquisite operating system for our cells? RNA — the single-stranded cousin of DNA. Scientists suspect that a better understanding of RNAs will allow us to more deeply understand healthy cells, and to design better treatments for those infected by disease. (See below for more RNA info.)

Now EteRNA, a collaborative online game, allows ordinary citizens to help biologists take a crack at solving a challenging RNA mystery, namely: what are the rules governing its folding? Players who assemble the best RNA designs online will see their creations synthesized in a Stanford biochemistry lab!

Drs. Adrien Treuille (Asst. Prof. of Computer Science at CMU) and Rhiju Das (Asst. Prof. of Biochemistry at Stanford) met while completing their postdoctoral research at the University of Washington, and collaborated on another online venture — FoldIt — aimed at understanding protein folding. Hoping that a similar approach could be used to crack the mysteries of RNA folding, they teamed up with doctoral student Jeehyung Lee to create a multi-player RNA-folding game. In an added twist, the RNA that players design is then synthesized in a Stanford biochemistry lab, to see if the folding pattern was indeed correct.

What’s so hard about RNA folding?

“Our computational models are not yet sophisticated enough to correctly predict when a particular RNA design will fold correctly in practice,” Treuille said in an email interview. “It is easy to create RNA designs which computers predict will fold properly, but which will not when synthesized. … We hope that the EteRNA community will be able to put forth a more complete set of hypotheses about when RNAs fold properly, and use these hypotheses to design a set of new RNA designs that fold into exotic, and ultimately medically useful shapes.”

Read the rest of this entry »