playgraph

Adobe InMarket to Close Down

David Sharek — Fri, 29 Jul 2011 16:01:06 +0000

Unfortunately, Adobe’s InMarket and AIR app store will close down permanently on 31 August 2011. This means that the NASA-TLX and HRI-Workload desktop applications will no longer be available from Adobe. Instead, the apps will be available in-house or through our Intel AppUp store. If you have installed either of the apps from Adobe, please install the new versions so that you will continue to be able to receive updates. Future plans for the apps include Android and Apple versions.

For more information on the closing of the Adobe InMarket store, click here: http://www.adobe.com/devnet/inmarket.html

Mechanical Turk Master Workers

David Sharek — Fri, 24 Jun 2011 22:18:26 +0000

Amazon recently released a new category of Worker on the Mechanical Turk. These workers are called Masters, and they are elite, established workers who have demonstrated accuracy and efficiency in the work they produce.Currently, there are two categories that a Worker can be a master in, Photo Moderation and Categorization. I imagine that more categories will emerge as Amazon continues to develop this service. Selecting to use a Master instead of a regular Worker will cost you an additional 20% on top of the reward amount you set. The Experiments with the Mechanical Turk tutorial has been updated with screenshots and a brief description of this new feature.

Experiments using Mechanical Turk. Part 2

David Sharek — Mon, 06 Jun 2011 15:56:09 +0000

Welcome to part 2 of a multi-part series on how to conduct scientific experiments using Amazon’s Mechanical Turk.

This tutorial will show you how to:

Set up an Amazon Mechanical Turk Requester Account.
Set up a HIT template that will link to your online experiment. You will need to have a pre-developed online experiment or questionnaire. To keep this introduction simple, a SurveyMonkey questionnaire is used.

Let’s get started!

If you have followed along from part 1 of this tutorial, you should have completed entering your information in the Enter Properties tab in the Create a new HIT Template page.
At the top of the he Create a new HIT Template page, click on the Design Layout tab.
This tab allows you to create the content that the worker’s will see when they click on your HIT. In it, we will create a:
1. Title
2. Description
3. Link to your experiment
4. Experiment completion code text field and submit button.
The design layout for this template defaults to the text “What is…” and provides a large text entry field. Select the “What is…” text and change it to the title of your experiment, for example, “My SurveyMonkey Experiment“. Now enter in some text below the title to explain what your experiment will require the workers to do. This part is very important as you do not want to surprise the workers with an experiment that requires more time or effort than they had anticipated. This could lead to them quitting mid-experiment. Some things to include are:
1. A description of the experiment or survey. What kind of questions will they have to answer? If they will participate in a task, what sort of task is it?
2. An estimate of how long the entire experiment should take them.
3. If the experiment requires the workers to have special skills or backgrounds, include those requirements here.
4. If the experiment requires specific computer requirements such as screen resolution, bandwidth, or plugin (e.g., Adobe Flash), enter them here and provide a link to the plugin download page.
Next, you will need to add the link for the experiment.
1. Under the experiment description, enter some text such as: “Click Here to Start the HIT“.
2. Now we need to turn this text into an actual link.
3. To do this, click on the Edit HTML Source button located at the right of the text editor toolbar.
4. You should now see some HTML code.
5. Locate the the text you just entered and add the following code in front of the text:
  <a href="http://www.surveymonkey.com/myLink" target="_blank">
6. Then, add the following code to the end of the text:
  </a>
7. don’t forget to change the URL (above) to match the link to your SurveyMonkey questionnaire.
Finally, we will need to create the experimental completion code section.
1. Click the Edit HTML Source button to exit the HTML editor mode.
2. On a new line, after the link you just created, enter a heading such as Completion Steps:.
3. Then, on a new line, enter the following instructions: Once you have completed the experiment, a completion code will be displayed. Please enter the completion code below to indicate that you have participated and completed the study so you can be paid.
4. Now let’s sort out the completion code text field. To do this, click the Edit HTML Source button to go back to the HTML editor mode.
5. Select the code:
  <p><textarea name="answer" cols="80" rows="3"></textarea></p>
  
  and replace it with this code:
  
  Completion Code: </span><input type="text" name="completionCode" id="completionCode" size="20" /></p>
6. Here is an example of how your HTML code should look like:
  <h1><strong>My SurveyMonkey Experiment</strong></h1>
  <ol>
  <li>You will be asked to answer a short survey on your beliefs of technology in the workplace.</li>
  <li>You will also be asked to answer a few questions about your usage of technology as well as some general demographic information. </li>
  <li>This should not take more than <b>20 minutes</b>. </li>
  </ol>
  <h2><a target="_blank" href="http://www.surveymonkey.com/myLink">Click Here to Start the HIT</a></h2>
  <h2><span style="color: rgb(51, 102, 255);">Completion Steps:</span></h2>
  <h3><span style="color: rgb(51, 51, 51);"><b>Once you have completed the experiment, a completion code will be displayed. </b></span><span style="color: rgb(51, 51, 51);"><b><br />
  Please enter the completion code below to indicate that you have participated and completed the study so you can be paid.</b></span></h3>
  <p>Completion Code: <input type="text" size="20" id="completionCode" name="completionCode" /></p>
7. Finally, click the Edit HTML Source button to exit the HTML editor mode.
8. You should now have successfully completed editing your HIT!
Click the Preview and Finish button at the bottom right of the page.
1. You should now see a preview of your HIT. Check it to make sure it looks and works as you expect. Don’t forget to click on the link to make sure it links to the correct survey.
2. If all is well, click on the Finish button at the bottom right of the page.
You should now be in the Publish page, and you should be able to see a line summarizing your newly created HIT.
1. Click the Select button to the left of the HIT you just created.
2. You will then be taken to a preview of your HIT where you can confirm that everything is in order.
3. If everything is to your liking, click on the Next button at the bottom right of the page.
You should now be in the Confirm and Publish Batch page.
1. Check that you have correctly entered all the HIT properties such as Number of HITs in this batch and Reward per Assignment
2. If all the information is correct, click the Publish HITs button to activate your HIT.
Congratulations! You have just published your first HIT!
You can watch your HIT being completed in real-time by clicking on the Manage tab at the top of the page. Click this button now.
1. You should be in the Manage Batches page.
2. Even though you probably don’t have any results just yet, click on the Results button at the top-right of your HIT.
3. You should be in the Review Results page and be able to see a table with columns such as HIT ID, Worker ID, and Completion Code..
4. Once your workers submit their completion code, you will be able to see it show up in this table.
5. If the completion code is correct, simply select the worker and approve them.

This bring us to the end of this tutorial. I hope it has been useful for you as you leverage the crowd for your experimental design needs. The next few tutorials in this series will cover: working with your IRB, setting up qualifications, designing flash-based experiments, and managing workers.

For a detailed investigation on the use of crowdsourcing for experimental design, check out the following article published in the journal, Behavior Research Methods:

Behrend, T., Sharek, D., Meade, A., & Wiebe, E. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 1-14. doi: 10.3758/s13428-011-0081-0

Please feel free to leave comments and questions below!

NASA-TLX and HRI-WM now available for FREE!

David Sharek — Fri, 25 Mar 2011 23:49:55 +0000

This is just a quick post to let everyone know that I have decided to release the NASA-TLX and the HRI-WM workload measures for free.

NASA-TLX now at the Intel AppUp Store!

David Sharek — Thu, 27 Jan 2011 16:43:53 +0000

Click here to get the NASA-TLX tool at the Intel AppUp Store.

The Intel AppUp center is a service that allows you to browse a catalog of applications for purchase or free download to your netbook or other personal computing device. Applications are available for many different categories including social, education, entertainment, communications etc. To check it out, go to http://www.appup.com and have a look around.

HRI Workload Tool Now at the Intel AppUp Store!

David Sharek — Thu, 27 Jan 2011 16:31:27 +0000

Click here to get the HRI-Workload Measurement tool at the Intel AppUp Store.

Human Factors Prize

David Sharek — Sat, 25 Dec 2010 16:23:24 +0000

The Human Factors and Ergonomics Society (HFES) has put together a Human Factors Prize aimed at recognizing excellence in human factors/ergonomics research. This is a new competition aimed at recruiting authors to publish in one of the top human factors journals aptly named Human Factors.

Submissions must be unpublished manuscripts based on the chosen theme for the competition. To kick things of, this year’s topic is health care ergonomics which is defined on the HFES website as “research at the intersection of health care and human factors. Suitable sample topics include human factors aspects of home health care, the ergonomics of laparoscopic equipment and procedures, patient care coordination, usability of electronic health records and informatics, macroergonomics of health care facilities, and use of simulation for health care training.”

This is a new competition with a $10,000 purse. Submissions are due 1 June 2011, and the winner will be announced on 15 August 2011.

Experiments using Mechanical Turk. Part 1

David Sharek — Sun, 19 Dec 2010 20:25:30 +0000

Welcome to part 1 of a multi-part series on how to conduct scientific experiments using Amazon’s Mechanical Turk.

This tutorial will show you how to:

Set up an Amazon Mechanical Turk Requester Account.
Set up a HIT template that will link to your online experiment. You will need to have a pre-developed online experiment or questionnaire. To keep this introduction simple, a SurveyMonkey questionnaire is used.

Let’s get started!

To begin, go to Amazon’s Mechanical Turk homepage.
Since you are going to be requesting participants, you need to pay attention to the right side of the screen that should look like this:
If you do not have an Amazon account, you will need to set one up. Click the Register Now link on the Mechanical Turk homepage or click here directly.
Once you have signed up and are signed in, click the Get Started button in the Requester section like in the image above.
You should now be in the Requester page (shown below). It is a bit cluttered, but full of good resources for learning how to use the Mechanical Turk effectively. We are going to ignore the majority of this page though, and focus on the tabs at the top of the page. Click the Design tab to begin.
Since we haven’t made any HITs yet, we need to start with the Design tab. Click it to get started.
On the Design HIT Templates page, you have the option of selecting a variety of sample HIT templates. Since we are simply going to be directing participants to a URL for a SurveyMonkey questionnaire, we don’t need a complicated template. Click on the Start with this Template button beside the Basic Open-ended Question template. It should be the first option in the Sample HIT Templates box.
The next page you will see is the Create a new HIT Template page with three main tabs:
While remaining in the Enter Properties tab, first enter a description of the template in the Template Name input field. This can be anything you want. I suggest giving it a name that describes the experiment, and a version number. You may want to tweak it later and use it for the same experiment, so including a versioning system is a good idea. Now let’s tackle the three main sections on this page starting with the Describe your HIT section:
For the Title, enter a succinct description of your experiment. For example, if you were investigating personality and attitudes, you could enter: Personality and Attitudes Survey. This is important because the Workers will be scanning these brief titles for HITs that they feel they can complete. If your title is too complex or “researchy” you may scare people off.
For the Description, you can include more information about your task. I still recommend keeping this section brief, accurate, and to the point. For example you could include something like: You are invited to participate in a research study. The goal of this study is to investigate personality and attitude differences. This tells Workers that the HIT is a research study and it explains what is being investigated.
For the Keywords, just enter keywords that best describe your HIT. Keep in mind the types of words people may be searching for. If you plan to conduct a lot of HITs, you may want to include your Requester name as well. I recommend avoiding sensationalistic words such as: quick, easy, simple etc. Let your description and payment amounts do the talking.Additionally, keep in mind your institution’s experimental guidelines policies. I’f you an an institutional review board approval process, I suggest including all of the content that you add to your HIT.
Now let’s begin exploring the options for gathering the Workers starting with the Working on your HIT section:For Time alloted per assignment, enter the amount of time that you want to give for Workers to complete the HIT once they accept it. For a 30 minute experiment, I typically set this value to one hour. I think there is a sweet spot here where you set the time to be long enough to be reasonable, but not too long so that Workers will feel compelled to complete the HIT in a timely manner. At this point, I think I should warn you that not all Workers accept HITs in the same way if the task is located on another website (like in this example). The correct procedure is to:
1. Read the HIT instructions
2. Accept the HIT
3. Complete the task
4. Indicate that the HIT has been completed
However, I have noticed that some workers will:
1. Read the HIT instructions (which contains the URL to the outside task)
2. Click on the URL
3. Complete the task
4. Go back to the HIT and accept it
5. Immediately indicate that they have completed it.
The problem with this second approach is that when you look at the Worker statistics, it appears that the Worker only took a few seconds to complete your 30 minute task.

For the HIT expires in value, I typically set this to seven days. Setting this value also requires a bit of balance. If you set the HIT to expire too soon, then you risk not recruiting enough participants. You will have to re-submit your HIT in this case which could throw off reporting and other data capture measures. However, if you set the HIT to expire too far into the future, then you run the risk of Workers choosing other HITs before yours since they know they have time.

Mechanical Turk Masters is a new option that was released in June 2011. A Mechanical Turk Master is an established and elite Worker who has provided consistently top-notch work in specific types of HITs. These Masters are highly accurate and good at what they do so they can demand top dollar. Currently, Masters can be selected for Photo Moderation tasks and Categorization tasks. Both of these tasks probably will not be useful for the typical scientific research project, although I am sure there are a few interesting research studies that could be developed to investigate the differences between Masters and regular Workers. If you do want to use Masters, then you will have to pay an additional 20% on top of the normal reward amount.

Additional Qualifications can be set in the final area in this section. I’ll list the possible options and give some examples of when and how to use some of them:
- Location: You can select which countries you want or do not want your participants to come from. I typically do not set this value since I want to include as many countries as possible. For an good look at the current demographics of Mechanical Turk users check out Panos Ipeirotis’s report. Keep in mind that currently, Amazon only pays cash to people with U.S. or Indian bank accounts. All other Workers are paid in Amazon gift cards.
- HIT approval Rate (%): I usually keep this at the default greater or equal to 95%. This means that Workers with a fairly high track record will only be allowed to participate.
- Adult Content Qualification: I don’t set this as my research doesn’t contain adult content. If yours does, then you would need to set this to required.
- Number of HITs Approved: I also don’t set this, but if you want your Workers to have already had a certain number of HITs approved, you can set this here.
- I always check the Required for preview checkbox. Since I often include a URL to my experiment, I don’t want Workers who don’t meet the qualifications to be able to view the experiment’s webpage.
The Paying Workers section is where you will set how much you want to pay each Worker, and how many people you want to participate in your experiment:

For the Reward per assignment section, I normally pay out about 80 cents for a 30 -40 minute experiment. You may be trying to calculate an hourly wage from this and determine that this is just way too low. To understand this you must first realize that the normal HIT lasts less than a minute and pays workers just a few cents per task. There are HITs that ask people to write articles, or product reviews which pay more, but in general a 30-40 minute task is still considered uncommon. Following that, a payment of 80 cents is also still quite uncommon and Workers seem to be eager for a change of pace. I have played with the reward amount bringing it down to 50 cents and I found that the quality of the data in the open-ended subjective responses seemed to be more rushed and less thoughtful compared to their 80 cent counterparts. Additionally, it took a full day longer to recruit 300 Workers; normally I can get 300 Workers in a day.

Set the Number of assignments per HIT to the number of workers that you want to participate in your experiment.

For the Results are automatically approved in section, I just leave the values at the default 7 days. You really shouldn’t ever let HITs be auto approved, but for the Worker’s peace of mind, you should set this to a reasonable time so they won’t feel that they could be taken advantage of.
Click the Save button to save your work on this page.
Part 2 of this series tackles the Design Layout and the Preview and Finish tabs.

Open Exhibits: Interactive Gestures

David Sharek — Fri, 17 Dec 2010 17:04:47 +0000

Open Exhibits is a multitouch open source architecture supported by the National Science Foundation. They provide various modules and templates for displaying interactive information geared towards educators, museum exhibit designers, and other types of information visualization designers. You can download their core for free if you are a student, educator, or museum professional. Since Open Exhibits is based on GestureWorks, you would have to pay around $250 for a single license if you want to you use it for commercial purposes. It may well be worth it as it provides over 200 available gestures and limitless multitouch authoring for Flash, Flex, and Air development. If you have been thinking of researching or developing multitouch technologies but have been unsure of where to start, this very well may be the toolkit you have been waiting for.

Word Lens: Real-Time Translator App

David Sharek — Fri, 17 Dec 2010 14:10:43 +0000

This is a brilliant idea for an augmented reality app. The idea is simple: point your iPhone camera towards any object with printed text and the app translates the text and maps it back onto the object on your screen. Could you imagine if this was integrated into a pair of glasses? One of the reviewers on the App Store quite appropriately summed up this idea by quoting Arthur C. Clarke: “Any sufficiently advanced technology is indistinguishable from magic”.

Currently the app only works with English and French but apparently the two-man development team at Quest Visual is working to add more languages. Hopefully an Android version will also be available in the very near future.