This article explores the viability of online crowdsourcing for creating matched-comparison groups. This exploratory study compares survey results from a randomized control group to survey results from a matched-comparison group created from Amazon.com’s MTurk crowdsourcing service to determine their comparability. Study findings indicate that online crowdsourcing, a process that allows access to many participants to complete specific tasks, is a potentially viable resource for evaluation designs where access to comparison groups, large budgets, and/or time are limited. The article highlights the strengths and limitations of the online crowdsourcing approach and describes ways that it could potentially be used in evaluation practice.
AzzamT.ChristieC. (2006). Using public databases to study relative program impact. Canadian Journal of Program Evaluation, 22, 57–68.
2.
BehrendT. S.SharekD. J.MeadeA. W.WiebeE. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800–813.
3.
BerinskyA. J.HuberG. A.LenzG. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368. doi:10.1093/pan/mpr057
4.
BloomH. S.MichalopoulosC.CarolynJ. H. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In BloomH. S. (Ed.), Learning more from social experiments (pp. 173–235). New York, NY: Russel Sage Foundation.
5.
BuhrmesterM. D.KwangT.GoslingS. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6, 3–5.
6.
ChandlerD.KapelnerA. (2010). Breaking monotony with meaning: Motivation in crowdsourcing markets. Chicago, IL: University of Chicago, Becker Center on Chicago Price Theory.
CranoW.BrewerM.LacA. (2001). Principles and methods of social research (2nd ed.). New York, NY: Psychology Press.
9.
CookT. D.ShadishW. R.WongV. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724–750.
10.
DehejiaR. H.WahbaS. (1999). Causal effect in non-experimental studies: Reevaluating the evaluation of training programs. Journal of American Statistical Association, 94, 1053–1062.
GuoS.FraserM. W. (2010). Propensity score analysis: Statistical methods and applications. Thousand Oaks, CA: Sage.
14.
HortonJ.ChiltonL. (2010). The labor economics of paid crowdsourcing. In D. C. Parkes (Ed.), Proceedings of the 11th ACM conference on electronic commerce (pp. 209–218). New York, NY: ACM.
15.
HortonJ.RandD.ZeckhauserR. (2010). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 4, 399–442.
KrautR.OlsonJ.BanajiM.BruckmanA.CohenJ.CouperM. (2004). Psychological research online: Report of Board of Scientific Affairs’ Advisory Group on the Conduct of Research on the Internet. American Psychologist, 59, 105.
18.
KuhG. D. (2009). The national survey of student engagement. New Directions for Institutional Research, 141, 5–20.
19.
MarshH. W. (1998). Simulation study of non-equivalent group-matching and regression discontinuity designs: Evaluation of gifted and talented programs. Journal of Experimental Education, 66, 163–192.
20.
PaolacciG.ChandlerJ.IpeirotisP. I. (2010). Running experiments on Amazon mechanical turk. Judgment and Decision Making, 5, 411–419.
21.
RosenbaumP.RubinD. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
22.
RossJ.IraniL.SilbermanM. S.ZaldivarA.TomlinsonB. (2010). Who are the crowdworkers? Shifting demographics in Mechnical Turk. In E. Mynatt, K. Edwards, & T. Rodden (Eds.), CHI EA ’10: Proceedings of the 28th of the international conference extended abstracts on human factors in computing systems (pp. 2863–2872). New York, NY: ACM.
23.
ScrivenM. (2010). Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation, 31, 105–117.
24.
ShadishW.CookT.CampbellD. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
25.
ShadishW. R.ClarkM. H.SteinerP. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103, 1334–1344.
26.
SmithH. (1997). Matching with multiple controls to estimate treatment effects in observational studies. Sociological Methodology, 27, 325–353.
27.
StuartE. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1–21.
28.
WinchesterS. (2003). The meaning of everything: The story of the Oxford English Dictionary. New York, NY: Oxford University Press.
