Experiments using Mechanical Turk. Part 2

Welcome to part 2 of a multi-part series on how to conduct scientific experiments using Amazon’s Mechanical Turk.

This tutorial will show you how to:

1. Set up an Amazon Mechanical Turk Requester Account.
2. Set up a HIT template that will link to your online experiment. You will need to have a pre-developed online experiment or questionnaire. To keep this introduction simple, a SurveyMonkey questionnaire is used.

Let’s get started!

1. If you have followed along from part 1 of this tutorial, you should have completed entering your information in the Enter Properties tab in the Create a new HIT Template page.
2. At the top of the he Create a new HIT Template page, click on the Design Layout tab.
3. This tab allows you to create the content that the worker’s will see when they click on your HIT. In it, we will create a:
   1. Title
   2. Description
   3. Link to your experiment
   4. Experiment completion code text field and submit button.
4. The design layout for this template defaults to the text “What is…” and provides a large text entry field. Select the “What is…” text and change it to the title of your experiment, for example, “My SurveyMonkey Experiment“. Now enter in some text below the title to explain what your experiment will require the workers to do. This part is very important as you do not want to surprise the workers with an experiment that requires more time or effort than they had anticipated. This could lead to them quitting mid-experiment. Some things to include are:
   1. A description of the experiment or survey. What kind of questions will they have to answer? If they will participate in a task, what sort of task is it?
   2. An estimate of how long the entire experiment should take them.
   3. If the experiment requires the workers to have special skills or backgrounds, include those requirements here.
   4. If the experiment requires specific computer requirements such as screen resolution, bandwidth, or plugin (e.g., Adobe Flash), enter them here and provide a link to the plugin download page.
5. Next, you will need to add the link for the experiment.
   1. Under the experiment description, enter some text such as: “Click Here to Start the HIT“.
   2. Now we need to turn this text into an actual link.
   3. To do this, click on the Edit HTML Source button located at the right of the text editor toolbar.
   4. You should now see some HTML code.
   5. Locate the the text you just entered and add the following code in front of the text:
      <a href="http://www.surveymonkey.com/myLink" target="_blank">
   6. Then, add the following code to the end of the text:
      </a>
   7. don’t forget to change the URL (above) to match the link to your SurveyMonkey questionnaire.
6. Finally, we will need to create the experimental completion code section.
   1. Click the Edit HTML Source button to exit the HTML editor mode.
   2. On a new line, after the link you just created, enter a heading such as Completion Steps:.
   3. Then, on a new line, enter the following instructions: Once you have completed the experiment, a completion code will be displayed. Please enter the completion code below to indicate that you have participated and completed the study so you can be paid.
   4. Now let’s sort out the completion code text field. To do this, click the Edit HTML Source button to go back to the HTML editor mode.
   5. Select the code:
      <p><textarea name="answer" cols="80" rows="3"></textarea></p>

      and replace it with this code:

      Completion Code: </span><input type="text" name="completionCode" id="completionCode" size="20" /></p>
   6. Here is an example of how your HTML code should look like:
      <h1><strong>My SurveyMonkey Experiment</strong></h1>
      <ol>
          <li>You will be asked to answer a short survey on your beliefs of technology in the workplace.</li>
          <li>You will also be asked to answer a few questions about your usage of technology as well as some general demographic information.&nbsp;</li>
          <li>This should not take more than <b>20 minutes</b>.&nbsp;</li>
      </ol>
      <h2><a target="_blank" href="http://www.surveymonkey.com/myLink">Click Here to Start the HIT</a></h2>
      <h2><span style="color: rgb(51, 102, 255);">Completion Steps:</span></h2>
      <h3><span style="color: rgb(51, 51, 51);"><b>Once you have completed the experiment, a completion code will be displayed. </b></span><span style="color: rgb(51, 51, 51);"><b><br />
      Please enter the completion code below to indicate that you have participated and completed the study so you can be paid.</b></span></h3>
      <p>Completion Code:  <input type="text" size="20" id="completionCode" name="completionCode" /></p>
   7. Finally, click the Edit HTML Source button to exit the HTML editor mode.
   8. You should now have successfully completed editing your HIT!
7. Click the Preview and Finish button at the bottom right of the page.
   1. You should now see a preview of your HIT. Check it to make sure it looks and works as you expect. Don’t forget to click on the link to make sure it links to the correct survey.
   2. If all is well, click on the Finish button at the bottom right of the page.
8. You should now be in the Publish page, and you should be able to see a line summarizing your newly created HIT.
   1. Click the Select button to the left of the HIT you just created.
   2. You will then be taken to a preview of your HIT where you can confirm that everything is in order.
   3. If everything is to your liking, click on the Next button at the bottom right of the page.
9. You should now be in the Confirm and Publish Batch page.
   1. Check that you have correctly entered all the HIT properties such as Number of HITs in this batch and Reward per Assignment
   2. If all the information is correct, click the Publish HITs button to activate your HIT.
10. Congratulations! You have just published your first HIT!
    
11. You can watch your HIT being completed in real-time by clicking on the Manage tab at the top of the page. Click this button now.
    1. You should be in the Manage Batches page.
    2. Even though you probably don’t have any results just yet, click on the Results button at the top-right of your HIT.
    3. You should be in the Review Results page and be able to see a table with columns such as HIT ID, Worker ID, and Completion Code..
    4. Once your workers submit their completion code, you will be able to see it show up in this table.
    5. If the completion code is correct, simply select the worker and approve them.

This bring us to the end of this tutorial. I hope it has been useful for you as you leverage the crowd for your experimental design needs. The next few tutorials in this series will cover: working with your IRB, setting up qualifications, designing flash-based experiments, and managing workers.



For a detailed investigation on the use of crowdsourcing for experimental design, check out the following article published in the journal, Behavior Research Methods:

Behrend, T., Sharek, D., Meade, A., & Wiebe, E. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 1-14. doi: 10.3758/s13428-011-0081-0

Please feel free to leave comments and questions below!