IST 597 Special Topic (Srping 2019)

Crowdsourcing & Crowd-AI Systems

Logistic

  1. Please bring a laptop to class.
  2. Please finish each week's required reading (2 to 3 papers) before class.
  3. We will have a quiz, containing 2 to 3 simple questions about that week's reading, at the beginning of each class. These quizzes must be completed while in class and without consultation with online resources.
  4. If you cheat, you fail the course. You are allowed to discuss assignments with others, but nothing written (on paper, in electronic format, on whiteboards, etc) should exist after these discussions.

Syllabus

Date Lecture Readings & Assignments Student Presentation
1 1/10

Overview [Slides]

  • Introduction
  • Objectives, scope and logistic of the course
  • Discussion
  • Amazon Mechanical Turk demo
 -
2 1/17

Collective Intelligence, Crowdsourcing, and Human Computation [Slides]

  • What is Collective Intelligence
  • How to use the wisdom of crowds?
  • Types of collective intelligence
  • Definition of Human Computation

Readings (Please finish before class):

  1. Human-Computer Interaction and Collective Intelligence. Bigham et al. 2014.
  2. Human Computation: A Survey and Taxonomy of a Growing Field. Quinn et al. CHI'11.
 -
3 1/24 Class cancelled due to snow.

Readings (Please finish before class):

  1. Being a turker. Martin et al. CSCW'14.
  2. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. Hara et al. CHI'18.
  3. Demographics and Dynamics of Mechanical Turk Workers. Difallah et al. WSDM'18.
 Class cancelled due to snow.
4 1/31 Class cancelled due to snow.
  • Assignment 1 Due

Readings (Please finish before class):

  1. Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. Chris Callison-Burch. EMNLP'09.
  2. ImageNet: A large-scale hierarchical image database. Deng et al. CVPR'09.
  3. A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories. Mostafazadeh et al. NAACL'16.
 Class cancelled due to snow.
5 2/7

Workers & Wages [Slides]

  • Why would they work for pennies?
  • How much do workers earn?
  • How big is Amazon Mechanical Turk?
  • Who are the workers?

Good Human Annotations & How to Get them [Slides]

  • Usability in Crowdsourcing
  • Data Annotation for AI

Readings (Please finish before class):

  1. Utility of human-computer interactions: toward a science of preference measurement. Toomim et al. CHI'11.
6 2/14

Human Computation Workflow [Slides]

  • Why?
  • How does it work?
  • Is it cheaper or faster?
  • Crowd-Machine Workflow

Readings (Please finish before class):

  1. TurKit: human computation algorithms on mechanical turk. Little et al. UIST'10.
  2. CrowdForge: Crowdsourcing Complex Work. Kittur et al. UIST'11.
  3. Cascade: crowdsourcing taxonomy creation. Chilton et al. CHI'13.

Paper: Platemate: crowdsourcing nutritional analysis from food photographs. Noronha et al. UIST'11.

Presenter: Shih-Hong Huang

7 2/21

Real-time Crowdsourcing [Slides]

  • Why Using Low-Latency Crowdsourcing?
  • The Deployed Chorus

Readings (Please finish before class):

  1. Crowds in two seconds: enabling realtime crowd-powered interfaces. Bernstein et al. UIST'11.
  2. VizWiz: nearly real-time answers to visual questions. Bigham et al. UIST'10.
  3. A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing Huang et al. HCOMP'17.

Assignment 2-1 Presentation
8 2/28

Consensus & Quality Control [Slides]

  • Consensus algorithms
  • How do we choose when everyone is making a decision?
  • Assignment 2-2 Due

Readings (Please finish before class):

  1. SQUARE: A Benchmark for Research on Computing Crowd Consensus. Sheshadri et al. HCOMP'13.
  2. Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers? Panos Ipeirotis. 2010.
9 3/7 Spring Break. No class.
10 3/14 Citizen Science & CrowdFunding [Slides]
  • Project Proposal Presentation

Readings (Please finish before class):

  1. eBird: A citizen-based bird observation network in the biological sciences. Sullivan et al. Biological Conservation, 142(10), 2282-2292. 2009.
  2. Predicting protein structures with a multiplayer online game. Cooper et al. Nature volume 466, pages 756–760 (05 August 2010)
  3. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Lintott et al. Monthly Notices of the Royal Astronomical Society, 389(3), 1179-1189.
 Project Proposal Presentation
11 3/21

Crowd Writing [Slides], By Chieh-Yang Huang

Readings (Please finish before class):

  1. Soylent: a word processor with a crowd inside. Bernstein et al. UIST'10.
  2. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. Hahn et al. CHI'16.
  3. WearWrite: Crowd-Assisted Writing from Smartwatches. Nebeling et al. CHI'16.
12 3/28 Learnersourcing, Selfsourcing, and Friendsourcing [Slides]
  • Project Checkpoint Presentation

Readings (Please finish before class):

  1. AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning
  2. Selfsourcing personal tasks
  3. Estimating the social costs of friendsourcing
 Project Checkpoint Presentation
13 4/4 Class cancelled by the guest lecturer. Class cancelled by the guest lecturer.
14 4/11

Challenges and Threats [Slides]

Readings (Please finish before class):

  1. Information extraction and manipulation threats in crowd-powered systems. Lasecki et al. CSCW'14.
  2. "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang et al. HCOMP'16.

Paper: Do Distance Learners Connect? (CHI 2019).

Presenter: Na Sun.
15 4/18 The Future of Crowd Work

Readings (Please finish before class):

  1. The future of crowd work. Kittur et al. CSCW'13.

Paper: UIST 2019 Submission (under review).

Presenter: Shih-Hong Huang.
16 4/25 Guest speaker (tentative): Dr. Yen-Chia Hsu (CMU) Project Final Presentation
17 5/2 Final Exam Week. No class.
  • Final Project Report Due

Many materials are borrowed and modified from Jeff's class and Chris' class Thank you!