The Experts below are selected from a list of 11265 Experts worldwide ranked by ideXlab platform
Chris Callisonburch - One of the best experts on this subject based on the ideXlab platform.
-
worker demographics and earnings on amazon Mechanical Turk an exploratory analysis
Human Factors in Computing Systems, 2019Co-Authors: Kotaro Hara, Kristy Milland, Abigail Adams, Saiph Savage, Benjamin V Hanrahan, Jeffrey P Bigham, Chris CallisonburchAbstract:Prior research reported that workers on Amazon Mechanical Turk (AMT) are underpaid, earning about $2/h. But the prior research did not investigate the difference in wage due to worker characteristics (e.g., country of residence). We present the first data-driven analysis on wage gap on AMT. Using work log data and demographic data collected via online survey, we analyse the gap in wage due to different factors. We show that there is indeed wage gap; for example, workers in the U.S. earn $3.01/h while those in India earn $1.41/h.
-
creating speech and language data with amazon s Mechanical Turk
North American Chapter of the Association for Computational Linguistics, 2010Co-Authors: Chris Callisonburch, Mark DredzeAbstract:In this paper we give an introduction to using Amazon's Mechanical Turk crowdsourcing platform for the purpose of collecting data for human language technologies. We survey the papers published in the NAACL-2010 Workshop. 24 researchers participated in the workshop's shared task to create data for speech and language applications with $100.
-
fast cheap and creative evaluating translation quality using amazon s Mechanical Turk
Empirical Methods in Natural Language Processing, 2009Co-Authors: Chris CallisonburchAbstract:Manual evaluation of translation quality is generally thought to be excessively time consuming and expensive. We explore a fast and inexpensive way of doing it using Amazon's Mechanical Turk to pay small sums to a large number of non-expert annotators. For $10 we redundantly recreate judgments from a WMT08 translation task. We find that when combined non-expert judgments have a high-level of agreement with the existing gold-standard judgments of machine translation quality, and correlate more strongly with expert judgments than Bleu does. We go on to show that Mechanical Turk can be used to calculate human-mediated translation edit rate (HTER), to conduct reading comprehension experiments with machine translation, and to create high quality reference translations.
Ben Bederson - One of the best experts on this subject based on the ideXlab platform.
-
error driven paraphrase annotation using Mechanical Turk
North American Chapter of the Association for Computational Linguistics, 2010Co-Authors: Olivia Buzek, Philip Resnik, Ben BedersonAbstract:The source text provided to a machine translation system is typically only one of many ways the input sentence could have been expressed, and alternative forms of expression can often produce a better translation. We introduce here error driven paraphrasing of source sentences: instead of paraphrasing a source sentence exhaustively, we obtain paraphrases for only the parts that are predicted to be problematic for the translation system. We report on an Amazon Mechanical Turk study that explores this idea, and establishes via an oracle evaluation that it holds the potential to substantially improve translation quality.
-
MTurk@HLT-NAACL - Error Driven Paraphrase Annotation using Mechanical Turk
2010Co-Authors: Olivia Buzek, Philip Resnik, Ben BedersonAbstract:The source text provided to a machine translation system is typically only one of many ways the input sentence could have been expressed, and alternative forms of expression can often produce a better translation. We introduce here error driven paraphrasing of source sentences: instead of paraphrasing a source sentence exhaustively, we obtain paraphrases for only the parts that are predicted to be problematic for the translation system. We report on an Amazon Mechanical Turk study that explores this idea, and establishes via an oracle evaluation that it holds the potential to substantially improve translation quality.
David Forsyth - One of the best experts on this subject based on the ideXlab platform.
-
utility data annotation with amazon Mechanical Turk
Computer Vision and Pattern Recognition, 2008Co-Authors: Alexander Sorokin, David ForsythAbstract:We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.
-
CVPR Workshops - Utility data annotation with Amazon Mechanical Turk
2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008Co-Authors: Alexander Sorokin, David ForsythAbstract:We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.
Klaus Zechner - One of the best experts on this subject based on the ideXlab platform.
-
using amazon Mechanical Turk for transcription of non native speech
North American Chapter of the Association for Computational Linguistics, 2010Co-Authors: Keelan Evanini, Derrick Higgins, Klaus ZechnerAbstract:This study investigates the use of Amazon Mechanical Turk for the transcription of non-native speech. Multiple transcriptions were obtained from several distinct MTurk workers and were combined to produce merged transcriptions that had higher levels of agreement with a gold standard transcription than the individual transcriptions. Three different methods for merging transcriptions were compared across two types of responses (spontaneous and read-aloud). The results show that the merged MTurk transcriptions are as accurate as an individual expert transcriber for the read-aloud responses, and are only slightly less accurate for the spontaneous responses.
Noah A Smith - One of the best experts on this subject based on the ideXlab platform.
-
rating computer generated questions with Mechanical Turk
North American Chapter of the Association for Computational Linguistics, 2010Co-Authors: Michael Heilman, Noah A SmithAbstract:We use Amazon Mechanical Turk to rate computer-generated reading comprehension questions about Wikipedia articles. Such application-specific ratings can be used to train statistical rankers to improve systems' final output, or to evaluate technologies that generate natural language. We discuss the question rating scheme we developed, assess the quality of the ratings that we gathered through Amazon Mechanical Turk, and show evidence that these ratings can be used to improve question generation.
-
MTurk@HLT-NAACL - Rating Computer-Generated Questions with Mechanical Turk
2010Co-Authors: Michael Heilman, Noah A SmithAbstract:We use Amazon Mechanical Turk to rate computer-generated reading comprehension questions about Wikipedia articles. Such application-specific ratings can be used to train statistical rankers to improve systems' final output, or to evaluate technologies that generate natural language. We discuss the question rating scheme we developed, assess the quality of the ratings that we gathered through Amazon Mechanical Turk, and show evidence that these ratings can be used to improve question generation.
