Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024
Amazon Mechanical Turk (MTurk) has been one of the most popular platforms for online research in psychology and the social sciences in general. While concerns about MTurk data quality have been raised, the platform continues to be widely used. The question is whether the MTurk platform is suitable f...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2025-07-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/10.1098/rsos.250361 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1839611645958029312 |
|---|---|
| author | Hagar Shimoni Vadim Axelrod |
| author_facet | Hagar Shimoni Vadim Axelrod |
| author_sort | Hagar Shimoni |
| collection | DOAJ |
| description | Amazon Mechanical Turk (MTurk) has been one of the most popular platforms for online research in psychology and the social sciences in general. While concerns about MTurk data quality have been raised, the platform continues to be widely used. The question is whether the MTurk platform is suitable for research and, if so, whether it is used optimally. We conducted a systematic investigation of MTurk data quality and reliability, including main and replication experiments, with more than 1300 participants subdivided into three cohorts: (i) workers (i.e. participants on the MTurk platform) with master requirement (i.e. high-performing workers selected by MTurk), (ii) workers without master requirement, and (iii) workers without master requirement, but with a 95% or above approval rate. We found that master workers almost never missed attentional checks, exhibited high reliability and showed no tendency towards straightlining, therefore, these workers are recommended, especially when the naivety of participants is not a strong prerequisite and no large sample size is required. In contrast, the workers without restrictions or with a 95% or above approval-rate threshold missed many attentional checks, exhibited low reliability and showed a tendency towards straightlining, raising serious concerns about the suitability of these workers for research. |
| format | Article |
| id | doaj-art-84766484cdf24e3882d5c6208a1d44e5 |
| institution | Matheson Library |
| issn | 2054-5703 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj-art-84766484cdf24e3882d5c6208a1d44e52025-07-28T15:20:45ZengThe Royal SocietyRoyal Society Open Science2054-57032025-07-0112710.1098/rsos.250361Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024Hagar Shimoni0Vadim Axelrod1The Gonda Multidisciplinary Brain Research Center, Bar-Ilan University, Ramat Gan, IsraelThe Gonda Multidisciplinary Brain Research Center, Bar-Ilan University, Ramat Gan, IsraelAmazon Mechanical Turk (MTurk) has been one of the most popular platforms for online research in psychology and the social sciences in general. While concerns about MTurk data quality have been raised, the platform continues to be widely used. The question is whether the MTurk platform is suitable for research and, if so, whether it is used optimally. We conducted a systematic investigation of MTurk data quality and reliability, including main and replication experiments, with more than 1300 participants subdivided into three cohorts: (i) workers (i.e. participants on the MTurk platform) with master requirement (i.e. high-performing workers selected by MTurk), (ii) workers without master requirement, and (iii) workers without master requirement, but with a 95% or above approval rate. We found that master workers almost never missed attentional checks, exhibited high reliability and showed no tendency towards straightlining, therefore, these workers are recommended, especially when the naivety of participants is not a strong prerequisite and no large sample size is required. In contrast, the workers without restrictions or with a 95% or above approval-rate threshold missed many attentional checks, exhibited low reliability and showed a tendency towards straightlining, raising serious concerns about the suitability of these workers for research.https://royalsocietypublishing.org/doi/10.1098/rsos.250361Amazon Mechanical TurkMTurkonline experimentsdata qualityreliabilitymaster workers |
| spellingShingle | Hagar Shimoni Vadim Axelrod Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024 Royal Society Open Science Amazon Mechanical Turk MTurk online experiments data quality reliability master workers |
| title | Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024 |
| title_full | Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024 |
| title_fullStr | Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024 |
| title_full_unstemmed | Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024 |
| title_short | Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024 |
| title_sort | assessing the quality and reliability of the amazon mechanical turk mturk data in 2024 |
| topic | Amazon Mechanical Turk MTurk online experiments data quality reliability master workers |
| url | https://royalsocietypublishing.org/doi/10.1098/rsos.250361 |
| work_keys_str_mv | AT hagarshimoni assessingthequalityandreliabilityoftheamazonmechanicalturkmturkdatain2024 AT vadimaxelrod assessingthequalityandreliabilityoftheamazonmechanicalturkmturkdatain2024 |