Playgraph

Experiments using Mechanical Turk. Part 1

Welcome to part 1 of a multi-part series on how to conduct scientific experiments using Amazon’s Mechanical Turk.

This tutorial will show you how to:

  1. Set up an Amazon Mechanical Turk Requester Account.
  2. Set up a HIT template that will link to your online experiment. You will need to have a pre-developed online experiment or questionnaire. To keep this introduction simple, a SurveyMonkey questionnaire is used.

Let’s get started!

  1. To begin, go to Amazon’s Mechanical Turk homepage.
  2. Since you are going to be requesting participants, you need to pay attention to the right side of the screen that should look like this:
  3. If you do not have an Amazon account, you will need to set one up. Click the Register Now link on the Mechanical Turk homepage or click here directly.
  4. Once you have signed up and are signed in, click the Get Started button in the Requester section like in the image above.
  5. You should now be in the Requester page (shown below). It is a bit cluttered, but full of good resources for learning how to use the Mechanical Turk effectively. We are going to ignore the majority of this page though, and focus on the tabs at the top of the page. Click the Design tab to begin.
  6. Since we haven’t made any HITs yet, we need to start with the Design tab. Click it to get started.
  7. On the Design HIT Templates page, you have the option of selecting a variety of sample HIT templates. Since we are simply going to be directing participants to a URL for a SurveyMonkey questionnaire, we don’t need a complicated template. Click on the Start with this Template button beside the Basic Open-ended Question template. It should be the first option in the Sample HIT Templates box.
  8. The next page you will see is the Create a new HIT Template page with three main tabs:
  9. While remaining in the Enter Properties tab, first enter a description of the template in the Template Name input field. This can be anything you want. I suggest giving it a name that describes the experiment, and a version number. You may want to tweak it later and use it for the same experiment, so including a versioning system is a good idea. Now let’s tackle the three main sections on this page starting with the Describe your HIT section:
  10. For the Title, enter a succinct description of your experiment. For example, if you were investigating personality and attitudes, you could enter: Personality and Attitudes Survey. This is important because the Workers will be scanning these brief titles for HITs that they feel they can complete. If your title is too complex or “researchy” you may scare people off.
  11. For the Description, you can include more information about your task. I still recommend keeping this section brief, accurate, and to the point. For example you could include something like: You are invited to participate in a research study. The goal of this study is to investigate personality and attitude differences. This tells Workers that the HIT is a research study and it explains what is being investigated.
  12. For the Keywords, just enter keywords that best describe your HIT. Keep in mind the types of words people may be searching for. If you plan to conduct a lot of HITs, you may want to include your Requester name as well. I recommend avoiding sensationalistic words such as: quick, easy, simple etc. Let your description and payment amounts do the talking.Additionally, keep in mind your institution’s experimental guidelines policies. I’f you an an institutional review board approval process, I suggest including all of the content that you add to your HIT.
  13. Now let’s begin exploring the options for gathering the Workers starting with the Working on your HIT section:For Time alloted per assignment, enter the amount of time that you want to give for Workers to complete the HIT once they accept it. For a 30 minute experiment, I typically set this value to one hour. I think there is a sweet spot here where you set the time to be long enough to be reasonable, but not too long so that Workers will feel compelled to complete the HIT in a timely manner. At this point, I think I should warn you that not all Workers accept HITs in the same way if the task is located on another website (like in this example). The correct procedure is to:
    1. Read the HIT instructions
    2. Accept the HIT
    3. Complete the task
    4. Indicate that the HIT has been completed

    However, I have noticed that some workers will:

    1. Read the HIT instructions (which contains the URL to the outside task)
    2. Click on the URL
    3. Complete the task
    4. Go back to the HIT and accept it
    5. Immediately indicate that they have completed it.

    The problem with this second approach is that when you look at the Worker statistics, it appears that the Worker only took a few seconds to complete your 30 minute task.

    For the HIT expires in value, I typically set this to seven days. Setting this value also requires a bit of balance. If you set the HIT to expire too soon, then you risk not recruiting enough participants. You will have to re-submit your HIT in this case which could throw off reporting and other data capture measures. However, if you set the HIT to expire too far into the future, then you run the risk of Workers choosing other HITs before yours since they know they have time.

    Mechanical Turk Masters is a new option that was released in June 2011. A Mechanical Turk Master is an established and elite Worker who has provided consistently top-notch work in specific types of HITs. These Masters are highly accurate and good at what they do so they can demand top dollar. Currently, Masters can be selected for Photo Moderation tasks and Categorization tasks. Both of these tasks probably will not be useful for the typical scientific research project, although I am sure there are a few interesting research studies that could be developed to investigate the differences between Masters and regular Workers. If you do want to use Masters, then you will have to pay an additional 20% on top of the normal reward amount.


    Additional Qualifications
    can be set in the final area in this section. I’ll list the possible options and give some examples of when and how to use some of them:

    • Location: You can select which countries you want or do not want your participants to come from. I typically do not set this value since I want to include as many countries as possible. For an good look at the current demographics of Mechanical Turk users check out Panos Ipeirotis’s report. Keep in mind that currently, Amazon only pays cash to people with U.S. or Indian bank accounts. All other Workers are paid in Amazon gift cards.
    • HIT approval Rate (%): I usually keep this at the default greater or equal to 95%. This means that Workers with a fairly high track record will only be allowed to participate.
    • Adult Content Qualification: I don’t set this as my research doesn’t contain adult content. If yours does, then you would need to set this to required.
    • Number of HITs Approved: I also don’t set this, but if you want your Workers to have already had a certain number of HITs approved, you can set this here.
    • I always check the Required for preview checkbox. Since I often include a URL to my experiment, I don’t want Workers who don’t meet the qualifications to be able to view the experiment’s webpage.
  14. The Paying Workers section is where you will set how much you want to pay each Worker, and how many people you want to participate in your experiment:

    For the Reward per assignment section, I normally pay out about 80 cents for a 30 -40 minute experiment. You may be trying to calculate an hourly wage from this and determine that this is just way too low. To understand this you must first realize that the normal HIT lasts less than a minute and pays workers just a few cents per task. There are HITs that ask people to write articles, or product reviews which pay more, but in general a 30-40 minute task is still considered uncommon. Following that, a payment of 80 cents is also still quite uncommon and Workers seem to be eager for a change of pace. I have played with the reward amount bringing it down to 50 cents and I found that the quality of the data in the open-ended subjective responses seemed to be more rushed and less thoughtful compared to their 80 cent counterparts. Additionally, it took a full day longer to recruit 300 Workers; normally I can get 300 Workers in a day.Set the Number of assignments per HIT to the number of workers that you want to participate in your experiment.

    For the Results are automatically approved in section, I just leave the values at the default 7 days. You really shouldn’t ever let HITs be auto approved, but for the Worker’s peace of mind, you should set this to a reasonable time so they won’t feel that they could be taken advantage of.

  15. Click the Save button to save your work on this page.
  16. Part 2 of this series tackles the Design Layout and the Preview and Finish tabs.

7 Comments

  1. Rene says:

    Question 1
    How and where do you insert the URL to your experiment in Mechanical Turk? I would like to understand how this is done before I decide whether or not I will go ahead and use Mechanical Turk to link to a Web site that contains my online experiment.

    Question 2
    Can Mechanical Turk pass any information to the Web site that contains your online experiment? If not, is there a way for the users (the “workers”) on Mechanical Turk to create their own unique identifier (like a “user code”) on Mechanical Turk which they would then also enter into your online experiment in order to help you track which Mechanical Turk worker was which experimental participant in your separate online questionnaire?

    Thank you! :)

    • David Sharek says:

      Hi Rene,
      Your first question will be answered in the second part of the tutorial, which should be up shortly!

      To answer your second question, the mTurk does not have a mechanism to send worker info to the online experiment. This is mainly due to privacy concerns.

      At one time, the worker ID was visible in the URL, and I wrote a script to pull it down and send it to a database. This was so I could prevent workers from participating in different versions of an experiment multiple times. However, even in this case, I did not attach their IDs to their responses so anonymity was still maintained. Sadly this ‘feature’ has been removed and it appears to no longer be possible.

      Oftentimes, I actually create a unique ID for the workers once they reach my experiment. One approach is to ask them to enter a username which is then converted to a MD5 hash string ID.

      Finally, you can always create a unique ID at the end of your survey and ask the worker to enter it back into the mTurk. This way you can match participant with worker. I’m not sure how SurveyMonkey would handle that as I generally develop in Flash but there must be a way.

  2. [...] of a regular Worker will cost you an additional 20% on top of the reward amount you set. The Experiments with the Mechanical Turk tutorial has been updated with screenshots and a brief description of this new [...]

  3. [...] Click here for a tutorial I created for conducting research using Amazon's Mechanical Turk. [...]

  4. Faray says:

    How to create multiple pages in MT?

  5. Dan says:

    The Master’s qualification is just Amazon’s way of making 20% more money off of novice job posters.

    People who aren’t fluent in English have the Master’s qualification, people who have extremely high rejection rates have the Master’s qualification, people who SCAM have the Master’s qualification.

    The 18th century Mechanical Turk was a hoax, and so is Amazon’s “Masters” qualification game.

    There are tens of thousands of qualified people contracting out their services on Amazon’s job board. Amazon claims 20K are some sort of “Masters” that job posters should be willing to pay an extra 20% for.

    The contractors themselves don’t know why Amazon decided to call them Masters!

    What a deception!

    Wolfgang von Kempelen would be proud of Amazon!

  6. Dan says:

    The contractors don’t get any of that extra 20%.

    NOPE! That all goes in Amazon’s pocket!

    Imagine that you host a job board, and you claim that 20,000 of the people who contract their services out on your job board are “Masters” after having performed some work for you, unknowingly, when you used a pseudonym instead of your own name.

    You send those contractors an email stating that they are “Masters”, and now have access to jobs available only to “Masters”.

    Meanwhile, you’re telling people who post jobs on your board that unless they opt out, their work will only be available to “Masters”.

    You also tell them that because “Masters” are an “elite group”, you should expect to pay them more.

    Finally, you tell them that they have to pay you 20% more to have “Masters” only work as contractors on the jobs you post!

Leave a Reply