Assignment 2-1: Design a Worker Interface (10%)

(This assignment was developed based on Jeff's class project 2.)

In this assignment, you'll take on the role of a requester by creating tasks on Amazon Mechanical Turk, and submitting your first HIT "programmatically" -- that is, without using the GUI provided on the Amazon Mechanical Turk site. This is one of the first steps of really "programming the crowd" because it demonstrates the start of being able to write programs that automatically draw in human intelligence when it's needed.

A HIT (or task page) is just a web page, so we'll start by creating a web page for your task. A web page with that contains a form that will be submitted to Mechanical Turk with a specific format. You might find it useful to include the script "mturk.js" that is also included on that site. Similar to the Javascript that we developed in class, that script will help to set your page up for Amazon Mechanical Turk.

Task for the Workers: Write a Short Story Based on a Photo Sequence

The task you'll have workers perform is to write a short story (about 5 sentences) based on the following photo sequence. You need to figure out a reasonable interface design that encourages workers to write a good story. The interface should also be able to detect some trivial human errors, e.g. typos, or story being too short.

  1. https://crowd.ist.psu.edu/VIST/test_by_album_story_only/72157620996064968/3694350550.jpg
  2. https://crowd.ist.psu.edu/VIST/test_by_album_story_only/72157620996064968/3691598980.jpg
  3. https://crowd.ist.psu.edu/VIST/test_by_album_story_only/72157620996064968/3694351198.jpg
  4. https://crowd.ist.psu.edu/VIST/test_by_album_story_only/72157620996064968/3690803821.jpg
  5. https://crowd.ist.psu.edu/VIST/test_by_album_story_only/72157620996064968/3694352398.jpg

Creating the Web Page for your HIT

At minimum your web page needs to show the image sequence and provide a text box so that workers can enter the story.

The challenge of the assignment is to design an interface that collects high-quality data from workers. There is no set guidelines on what you need to include in your web page for collecting HITs, but you might consider some of the lessons we've learned in lecture and from the readings. For instance, how can the interface encourage high-quality responses? How can you make it as difficult to do this job poorly as to do it well? Could you use a gold-standard question for sanity check? How can you write your instructions clearly? Can you build in some other quality control mechanism? Maybe you can pay them a high amount, or maybe you will bonus them if they agree with others?

You will be graded in part on the quality of the collected stories, so put time and thought into how to encourage them to give you high-quality input.

You will get a chance for feedback on your interface from your classmates on 2/21. You'll need to have a prototype ready on the class day in order to get feedback.

Preparing Your Web Page for Mechanical Turk

We suggest hosting your HITs on Github Pages. The reason is that it's free, relatively easy, and supports https, which Mechanical Turk requires. It's not fancy web hosting. You'll have ~zero control over the backend. But, as we’ll see, almost everything we'll be doing is on the front-end anyway.

There's a good walk-through of how to do this on the Github Pages page: https://pages.github.com.

Or you can use PSU web page. It also comes with HTTPS.

What to turn in

In the class on 2/21, 2019, you need to present your prototype webpage in front of your classmates. The presentation should include the following elements:
  1. Open your interface web page from an URL with HTTPS.
  2. Walk through the process of (1) reading the instruction, (2) seeing the photos, (3) writing a story, and (4) submitting the HIT.
  3. Briefly introduce your design and why it can encourage good data quality.
Your classmates will be required to give you some feedback based on your presentation.