DPTE – the blog!

Dynamic Process Tracing Environment User Discussion

DPTE and Amazon Mechanical Turk

Recently some folks have begun to use Amazon Mechanical Turk (mturk.com) as a means of getting subjects to run through DPTE experiments. This can be a quick and easy way to get a lot of subjects who fall outside the traditional “college sophomore” demographic. If you are doing this, we’d appreciate hearing about it, and in particular, appreciate you posting any tips you might have.

We do know that the system may become overwhelmed if hundreds of subjects attempt to access it at the same time, so one significant tip is to batch your HITS on Mturk so that you limit the number who can access the system at the same time. We recommend batches of no more than 25-50 respondents. Schedule these so that one batch is completed before you release the next one. Unfortunately to date not batching in small numbers has resulted in losses of up to 10-15% of subjects who attempt to begin the study. Recent system updates should have helped with this, but batching is still a good idea.

One other suggestion is that you require Turkers to copy a completion code into the HIT using their subject ID assigned by DPTE. This will make it easier for you to verify that the code is valid. To do this you create an Announcement in which you show the system variable [[SYS_VAR_SUBJECT_ID]] on the announcement with text explaining what they should do and warning them not to close the announcement until they do. Make that the very last thing in your experiment and the whole experiment will close once they close the announcement.

Other ideas, experiences, and challenges?

One response to “DPTE and Amazon Mechanical Turk

  1. dynamicprocesstracing July 31, 2012 at 1:13 pm

    I have been doing some testing, and it looks like the sweet spot is currently around 35 subjects. More than that at one time seems to have significant system impact. I have been doing a Mechanical Turk based study and am releasing batches of 35 at a time with few problems.

    We do hope the system upgrade scheduled for August 1 will improve this and allow larger batches to run successfully.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: