IST 597 Special Topic (Srping 2019)

Crowdsourcing & Crowd-AI Systems


  1. Please bring a laptop to class.
  2. Please finish each week's required reading (2 to 3 papers) before class.
  3. We will have a quiz, containing 2 to 3 simple questions about that week's reading, at the beginning of each class. These quizzes must be completed while in class and without consultation with online resources.
  4. If you cheat, you fail the course. You are allowed to discuss assignments with others, but nothing written (on paper, in electronic format, on whiteboards, etc) should exist after these discussions.


Date Lecture Readings & Assignments Student Presentation
1 1/10

Overview [Slides]

  • Introduction
  • Objectives, scope and logistic of the course
  • Discussion
  • Amazon Mechanical Turk demo
2 1/17

Collective Intelligence, Crowdsourcing, and Human Computation [Slides]

  • What is Collective Intelligence
  • How to use the wisdom of crowds?
  • Types of collective intelligence
  • Definition of Human Computation

Readings (Please finish before class):

  1. Human-Computer Interaction and Collective Intelligence. Bigham et al. 2014.
  2. Human Computation: A Survey and Taxonomy of a Growing Field. Quinn et al. CHI'11.
3 1/24 Class cancelled due to snow.

Readings (Please finish before class):

  1. Being a turker. Martin et al. CSCW'14.
  2. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. Hara et al. CHI'18.
  3. Demographics and Dynamics of Mechanical Turk Workers. Difallah et al. WSDM'18.
Class cancelled due to snow.
4 1/31 Class cancelled due to snow.
  • Assignment 1 Due

Readings (Please finish before class):

  1. Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. Chris Callison-Burch. EMNLP'09.
  2. ImageNet: A large-scale hierarchical image database. Deng et al. CVPR'09.
  3. A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories. Mostafazadeh et al. NAACL'16.
Class cancelled due to snow.
5 2/7

Workers & Wages [Slides]

  • Why would they work for pennies?
  • How much do workers earn?
  • How big is Amazon Mechanical Turk?
  • Who are the workers?

Good Human Annotations & How to Get them [Slides]

  • Usability in Crowdsourcing
  • Data Annotation for AI

Readings (Please finish before class):

  1. Utility of human-computer interactions: toward a science of preference measurement. Toomim et al. CHI'11.
6 2/14

Human Computation Workflow [Slides]

  • Why?
  • How does it work?
  • Is it cheaper or faster?
  • Crowd-Machine Workflow

Readings (Please finish before class):

  1. TurKit: human computation algorithms on mechanical turk. Little et al. UIST'10.
  2. CrowdForge: Crowdsourcing Complex Work. Kittur et al. UIST'11.
  3. Cascade: crowdsourcing taxonomy creation. Chilton et al. CHI'13.

Paper: Platemate: crowdsourcing nutritional analysis from food photographs. Noronha et al. UIST'11.

Presenter: Shih-Hong Huang

7 2/21

Real-time Crowdsourcing: Sync With Users

  • Assignment 2-1 Due: Presentation

Readings (Please finish before class):

  1. Crowds in two seconds: enabling realtime crowd-powered interfaces. Bernstein et al. UIST'11.
  2. VizWiz: nearly real-time answers to visual questions. Bigham et al. UIST'10.
  3. A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Huang et al. HCOMP'17.

Assignment 2-1 Presentation

8 2/28

Real-time Crowdsourcing: Sync With Fast-Paced Systems

Assignment 2 Presentation
9 3/7 Spring Break. No class.
10 3/14

Crowd Writing, By Chieh-Yang Huang

Project Proposal Presentation
11 3/21 Citizen Science & CrowdFunding [Present Papers]
12 3/28 Learnersourcing, Selfsourcing, and Friendsourcing
[Present Papers]
13 4/4 Disagreement and Ambiguity Project Checkpoint Presentation
14 4/11

Challenges and Threats

[Present Papers]
15 4/18

(Buffer week. Potential guest speaker.)

16 4/25 Guest speaker (tentative): Dr. Yen-Chia Hsu (CMU) Project Final Presentation
17 5/2 Final Exam Week. No class.
  • Final Project Report Due

Many materials are borrowed and modified from Jeff's class and Chris' class Thank you!