IST 402: Crowdsourcing & Crowd-AI Systems

IST 402: Emerging Issues and Technologies (Spring 2020)

Crowdsourcing & Crowd-AI Systems

Course number: IST 402 - Students from all majors are welcome!
Instructor: Kenneth Huang (txh710@psu.edu)
Time: Spring 2020, Tue/Thur 10:35am - 11:50am
Course Location: Boucke Bldg 112
TA: Chieh-Yang Huang (czh5679@psu.edu)
Virtual Office Hours (Please check your email for the Zoom link):
- Prof. Kenneth Huang: Tuesday & Thursday 11:30am - 12:00pm
- Chieh-Yang Huang (TA): Monday & Friday 3pm - 4pm

Office Hours:

Prof. Kenneth Huang: Tuesday 4pm - 5pm, Westgate E357

Chieh-Yang Huang (TA): Monday & Friday 3pm - 4pm, Westgate 2F Seating Area (IST side, next to Cafe)

Logistic for the Remote Period

All the lectures during the remote period will be pre-recorded. The video will be available online to the students before each meeting time of the class.
No quizzes during the remote period.

Logistic

~~Please bring a laptop (or a smartphone) to class.~~

Please finish each day's required reading (1 to 2 papers) before class.

We will have a quiz, containing 2 to 3 simple questions about that week's reading, at the beginning of each class. These quizzes must be completed while in class and without consultation with online resources.

If you cheat, you fail the course. You are allowed to discuss assignments with others, but nothing written (on paper, in electronic format, on whiteboards, etc) should exist after these discussions.

Syllabus

The blue background highlights the Remote Period, when all the activities (lecture, presentation, meetings, etc) are moved to online.

Date	Topics	Required Readings (Please Finish Before Class)
1/14 (Tue)	Overview [Slides] Assignment 0 Release (Please see Canvas for the due date.)
1/16 (Thu)	Collective Intelligence [Slides]	Human-Computer Interaction and Collective Intelligence. Bigham et al. 2014.
1/21 (Tue)	Human Computation [Slides] Assignment 1 Release (Please see Canvas for the due date.)	Human Computation: A Survey and Taxonomy of a Growing Field. Quinn et al. CHI'11.
1/23 (Thu)	Being a Crowd Worker [Slides]	Being a turker. Martin et al. CSCW'14. My MTurk (half) Workday. Jeffrey P. Bigham.
1/28 (Tue)	High-Quality Human Annotations & How to Get Them [Slides] Assignment 2-1 Release (Please see Canvas for the due date.)	Utility of human-computer interactions: toward a science of preference measurement. Toomim et al. CHI'11. Financial incentives and the "performance of crowds". Mason et al. HCOMP'09 .
1/30 (Thu)	Tutorial: Creating a Worker Interface Web Page, By Chieh-Yang Huang (TA) [Slides] [Code]	Evaluating and Improving the Usability of Mechanical Turk for Low-Income Workers in India. Khanna et al. ACM DEV'10
2/4 (Tue)	Human Computation Workflow [Slides]	TurKit: human computation algorithms on mechanical turk. Little et al. UIST'10. CrowdForge: Crowdsourcing Complex Work. Kittur et al. UIST'11.
2/6 (Thu)	(Nearly) Real-time Crowdsourcing [Slides] Final Project Release (Please see Canvas for the due date.)	Crowds in two seconds: enabling realtime crowd-powered interfaces. Bernstein et al. UIST'11. VizWiz: nearly real-time answers to visual questions. Bigham et al. UIST'10.
2/11 (Tue)	Assignment 2-1 Presentation (1)
2/13 (Thu)	Assignment 2-1 Presentation (2)
2/18 (Tue)	Crowd-Powered Conversational Assistant [Slides] Assignment 2-2 Release (Please see Canvas for the due date.)	Chorus: A Crowd-Powered Conversational Assistant. Lasecki et al. UIST'13. Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time. Huang et al. CHI'18.
2/20 (Thu)	Tutorial: Testing and Posting HITs to MTurk, By Chieh-Yang Huang (TA) [Slides] [Code]	We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers. Salehi et al. CHI'15.
2/25 (Tue)	Crowd Writing [Slides]	Soylent: a word processor with a crowd inside. Bernstein et al. UIST'10. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. Hahn et al. CHI'16.
2/27 (Thu)	Crowdfunding [Slides]	Crowdfunding: Motivations and deterrents for participation. Gerber et al. TOCHI'13.
3/3 (Tue)	Final Project Proposal Presentation (1)
3/5 (Thu)	Final Project Proposal Presentation (2)
3/10 (Tue)	Spring Break - No Classes
3/12 (Thu)	Spring Break - No Classes
3/17 (Tue)	The Remote Period starts. Preparation For Remote Teaching. No Lecture.
3/19 (Thu)	Selfsourcing, Friendsourcing, and Learnersourcing [Slides] [Video on Canvas]	Selfsourcing personal tasks. Teevan et al. CHI'14 EA. Estimating the social costs of friendsourcing. Rzeszotarski et al. CHI'14.
3/24 (Tue)	Citizen Science [Slides] [Video on Canvas]	eBird: A citizen-based bird observation network in the biological sciences. Sullivan et al. Biological Conservation, 142(10), 2282-2292. 2009. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Lintott et al. Monthly Notices of the Royal Astronomical Society, 389(3), 1179-1189.
3/26 (Thu)	Guest Lecture: Designing Interactive Systems for Community Citizen Science, By Yen-Chia Hsu (CMU) [Video on Canvas]	Community-Empowered Air Quality Monitoring System. Hsu et al. CHI'17. When Human-Computer Interaction Meets Community Citizen Science. Hsu et al. Communications of the ACM, February 2020, Vol. 63 No. 2, Pages 31-34.
3/31 (Tue)	Attacks and Threats in Crowdsourcing [Slides] [Video on Canvas]	Information extraction and manipulation threats in crowd-powered systems. Lasecki et al. CSCW'14. "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang et al. HCOMP'16.
4/2 (Thu)	Guest Lecture: A future of work in solidarity with workers, By Saiph Savage. [Video on YouTube] ~~[Canceled] Rock Ethics Institute Lippin Lecture: Automating Inequality, By Virginia Eubanks~~	Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth. Chiang, et al. CSCW'18. Becoming the Super Turker: Increasing Wages via a Strategy from High Earning Workers. Savage, et al. WWW'20. Optional Readings By Dr. Virginia Eubanks: We created poverty. Algorithms won't make that go away. The Guardian, Opinion. The Digital Poorhouse. Harper's Magazine.
4/7 (Tue)	Final Project Office Hour (No Lecture)
4/9 (Thu)	Final Project Office Hour (No Lecture)
4/14 (Tue)	Quality Control & Crowd Label Aggregation [Slides] [Video on Canvas]	SQUARE: A Benchmark for Research on Computing Crowd Consensus. Sheshadri et al. HCOMP'13. Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers? Panos Ipeirotis. 2010.
4/16 (Thu)	Guest Lecture: Human-AI Systems for Visual Information Access, By Anhong Guo (CMU) [Video on Canvas]	Facade: Auto-generating Tactile Interfaces to Appliances. Guo et al. CHI'17. StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible. Guo et al. UIST'19.
4/21 (Tue)	Designing Hybrid Intelligence: From Interaction to Co-Learning, By Yi-Ching (Janet) Huang (TU/e) [Video on Canvas]	Supporting ESL Writing by Prompting Crowdsourced Structural Feedback. Huang et al. HCOMP'17. Reflection before/after Practice: Learnersourcing for Drawing Support. Huang et al. CHI EA'18.
4/23 (Thu)	The Future of (Crowd) Work	The future of crowd work. Kittur et al. CSCW'13. Towards an AI-powered Future that Works for Vocational Workers Thakkar et al. CHI'20 .
4/28 (Tue)	Final Project Office Hour (No Lecture)
4/30 (Thu)	Final Project Office Hour (No Lecture)

Many materials are borrowed and modified from Jeff's class and Chris' class Thank you!