Human Computation and Crowdsourcing

• Human computation: a survey and taxonomy of a growing field. Quinn et al. CHI'11.
• Labeling images with a computer game. von Ahn et al. CHI'04.
• Designing games with a purpose. von Ahn et al. Commun. ACM 51, 8 (August 2008), 58-67.
• Counting with the Crowd. Marcus et al. VLDB Endowment'12.

Microtask Platforms

• Demographics of Mechanical Turk. Ipeirotis. 2010.
• Demographics and Dynamics of Mechanical Turk Workers. Difallah et al. WSDM'18.
• All Those Wasted Hours: On Task Abandonment in Crowdsourcing. Han et al. WSDM'19

Workers & Wages

• Being a turker. Martin et al. CSCW'14.
• The future of crowd work. Kittur et al. CSCW'13.
• A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. Hara et al. CHI'18.
• Stop citing Ross et al. 2010, “Who are the crowdworkers?” Six Silberman. 2015.
• Fair payment. Dynamo Wiki.

Human Annotation for AI

• Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. Chris Callison-Burch. EMNLP'09.

Human Computation Workflow

• TurKit: human computation algorithms on mechanical turk. Little et al. UIST'10.
• CrowdForge: Crowdsourcing Complex Work. Kittur et al. UIST'11.
• Cascade: crowdsourcing taxonomy creation. Chilton et al. CHI'13.

Crowd-Machine Systems

• Platemate: crowdsourcing nutritional analysis from food photographs. Noronha et al. UIST'11.
• Alloy: Clustering with Crowds and Computation. Chang et al. CHI'16.
• Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence. Ece Kamar. IJCAI'16.

Real-time Crowdsourcing: Sync With Users

• Crowds in two seconds: enabling realtime crowd-powered interfaces. Bernstein et al. UIST'11.
• Soylent: a word processor with a crowd inside. Bernstein et al. UIST'10.
• VizWiz: nearly real-time answers to visual questions. Bigham et al. UIST'10.
• A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Huang et al. HCOMP'17.

Real-time Crowdsourcing: Sync With Fast-Paced Systems

• CrowdDB: answering queries with crowdsourcing. Franklin et al. SIGMOD'11.
• Real-time captioning by groups of non-experts. Lasecki et al. UIST'12.
• Chorus: a crowd-powered conversational assistant. Lasecki et al. UIST'13.
• Evorus: A Crowd-powered Conversational Assistant That Automates Itself Over Time. Huang et al. CHI'18.
• Zensors: Adaptive, Rapidly Deployable, Human-Intelligent Sensor Feeds. Laput et al. CHI'15.
• Crowd-AI Camera Sensing in the Real World. Guo et al. IMWUT'18.

Crowd Writing

• The Knowledge Accelerator: Big Picture Thinking in Small Pieces. Hahn et al. CHI'16.
• WearWrite: Crowd-Assisted Writing from Smartwatches. Nebeling et al. CHI'16.

Crowd-Powered Design & Critique

• Shepherding the crowd yields better work. Dow et al. CSCW'12.
• CrowdCrit: crowdsourcing and aggregating visual design critique. Luther et al. CSCW'14.

Challenges and Threats

• Information extraction and manipulation threats in crowd-powered systems. Lasecki et al. CSCW'14.
• "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang et al. HCOMP'16.
• https://www.maxhuibai.com/blog/evidence-that-responses-from-repeating-gps-are-random
• Flip coin bias
• Entailmetn Dataset

• Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. Chang et al. CHI'17.
• MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. Drapeau et al. HCOMP'16.
• Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. Aroyo et al. WebSci'13.
• Learning from crowds. Raykar et al. JMLR'10.

• Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey
• LabintheWild: Conducting large-scale online experiments with uncompensated samples
• eBird: A citizen-based bird observation network in the biological sciences
• Predicting protein structures with a multiplayer online game

• AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning
• Selfsourcing personal tasks
• Estimating the social costs of friendsourcing

• Field trial of tiramisu: crowd-sourcing bus arrival times to spur co-design. Zimmerman1 et al. CHI'11.
• Community-Empowered Air Quality Monitoring System. Hsu et al. CHI'17.

• Quality management on Amazon Mechanical Turk.

• Conducting behavioral research on Amazon's Mechanical Turk. Mason et al. (2012) 44: 1.
• Running Experiments on Amazon Mechanical Turk. Paolacci et al. Judgment and Decision Making, Vol. 5, No. 5, August 2010.