Welcome to part 1 of a multi-part series on how to conduct scientific experiments using Amazon’s Mechanical Turk.
This tutorial will show you how to:
- Set up an Amazon Mechanical Turk Requester Account.
- Set up a HIT template that will link to your online experiment. You will need to have a pre-developed online experiment or questionnaire. To keep this introduction simple, a SurveyMonkey questionnaire is used.
Let’s get started!
- To begin, go to Amazon’s Mechanical Turk homepage.
- Since you are going to be requesting participants, you need to pay attention to the right side of the screen that should look like this:
- If you do not have an Amazon account, you will need to set one up. Click the Register Now link on the Mechanical Turk homepage or click here directly.
- Once you have signed up and are signed in, click the Get Started button in the Requester section like in the image above.
- You should now be in the Requester page (shown below). It is a bit cluttered, but full of good resources for learning how to use the Mechanical Turk effectively. We are going to ignore the majority of this page though, and focus on the tabs at the top of the page. Click the Design tab to begin.
- Since we haven’t made any HITs yet, we need to start with the Design tab. Click it to get started.
- On the Design HIT Templates page, you have the option of selecting a variety of sample HIT templates. Since we are simply going to be directing participants to a URL for a SurveyMonkey questionnaire, we don’t need a complicated template. Click on the Start with this Template button beside the Basic Open-ended Question template. It should be the first option in the Sample HIT Templates box.
- The next page you will see is the Create a new HIT Template page with three main tabs:
- While remaining in the Enter Properties tab, first enter a description of the template in the Template Name input field. This can be anything you want. I suggest giving it a name that describes the experiment, and a version number. You may want to tweak it later and use it for the same experiment, so including a versioning system is a good idea. Now let’s tackle the three main sections on this page starting with the Describe your HIT section:
- For the Title, enter a succinct description of your experiment. For example, if you were investigating personality and attitudes, you could enter: Personality and Attitudes Survey. This is important because the Workers will be scanning these brief titles for HITs that they feel they can complete. If your title is too complex or “researchy” you may scare people off.
- For the Description, you can include more information about your task. I still recommend keeping this section brief, accurate, and to the point. For example you could include something like: You are invited to participate in a research study. The goal of this study is to investigate personality and attitude differences. This tells Workers that the HIT is a research study and it explains what is being investigated.
- For the Keywords, just enter keywords that best describe your HIT. Keep in mind the types of words people may be searching for. If you plan to conduct a lot of HITs, you may want to include your Requester name as well. I recommend avoiding sensationalistic words such as: quick, easy, simple etc. Let your description and payment amounts do the talking.Additionally, keep in mind your institution’s experimental guidelines policies. I’f you an an institutional review board approval process, I suggest including all of the content that you add to your HIT.
- Now let’s begin exploring the options for gathering the Workers starting with the Working on your HIT section:For Time alloted per assignment, enter the amount of time that you want to give for Workers to complete the HIT once they accept it. For a 30 minute experiment, I typically set this value to one hour. I think there is a sweet spot here where you set the time to be long enough to be reasonable, but not too long so that Workers will feel compelled to complete the HIT in a timely manner. At this point, I think I should warn you that not all Workers accept HITs in the same way if the task is located on another website (like in this example). The correct procedure is to:
- Read the HIT instructions
- Accept the HIT
- Complete the task
- Indicate that the HIT has been completed
However, I have noticed that some workers will:
- Read the HIT instructions (which contains the URL to the outside task)
- Click on the URL
- Complete the task
- Go back to the HIT and accept it
- Immediately indicate that they have completed it.
The problem with this second approach is that when you look at the Worker statistics, it appears that the Worker only took a few seconds to complete your 30 minute task.
For the HIT expires in value, I typically set this to seven days. Setting this value also requires a bit of balance. If you set the HIT to expire too soon, then you risk not recruiting enough participants. You will have to re-submit your HIT in this case which could throw off reporting and other data capture measures. However, if you set the HIT to expire too far into the future, then you run the risk of Workers choosing other HITs before yours since they know they have time.
Mechanical Turk Masters is a new option that was released in June 2011. A Mechanical Turk Master is an established and elite Worker who has provided consistently top-notch work in specific types of HITs. These Masters are highly accurate and good at what they do so they can demand top dollar. Currently, Masters can be selected for Photo Moderation tasks and Categorization tasks. Both of these tasks probably will not be useful for the typical scientific research project, although I am sure there are a few interesting research studies that could be developed to investigate the differences between Masters and regular Workers. If you do want to use Masters, then you will have to pay an additional 20% on top of the normal reward amount.
Additional Qualifications can be set in the final area in this section. I’ll list the possible options and give some examples of when and how to use some of them:
- Location: You can select which countries you want or do not want your participants to come from. I typically do not set this value since I want to include as many countries as possible. For an good look at the current demographics of Mechanical Turk users check out Panos Ipeirotis’s report. Keep in mind that currently, Amazon only pays cash to people with U.S. or Indian bank accounts. All other Workers are paid in Amazon gift cards.
- HIT approval Rate (%): I usually keep this at the default greater or equal to 95%. This means that Workers with a fairly high track record will only be allowed to participate.
- Adult Content Qualification: I don’t set this as my research doesn’t contain adult content. If yours does, then you would need to set this to required.
- Number of HITs Approved: I also don’t set this, but if you want your Workers to have already had a certain number of HITs approved, you can set this here.
- I always check the Required for preview checkbox. Since I often include a URL to my experiment, I don’t want Workers who don’t meet the qualifications to be able to view the experiment’s webpage.
- The Paying Workers section is where you will set how much you want to pay each Worker, and how many people you want to participate in your experiment:
For the Reward per assignment section, I normally pay out about 80 cents for a 30 -40 minute experiment. You may be trying to calculate an hourly wage from this and determine that this is just way too low. To understand this you must first realize that the normal HIT lasts less than a minute and pays workers just a few cents per task. There are HITs that ask people to write articles, or product reviews which pay more, but in general a 30-40 minute task is still considered uncommon. Following that, a payment of 80 cents is also still quite uncommon and Workers seem to be eager for a change of pace. I have played with the reward amount bringing it down to 50 cents and I found that the quality of the data in the open-ended subjective responses seemed to be more rushed and less thoughtful compared to their 80 cent counterparts. Additionally, it took a full day longer to recruit 300 Workers; normally I can get 300 Workers in a day.Set the Number of assignments per HIT to the number of workers that you want to participate in your experiment.
For the Results are automatically approved in section, I just leave the values at the default 7 days. You really shouldn’t ever let HITs be auto approved, but for the Worker’s peace of mind, you should set this to a reasonable time so they won’t feel that they could be taken advantage of.
- Click the Save button to save your work on this page.
- Part 2 of this series tackles the Design Layout and the Preview and Finish tabs.