Sage Journals: Discover world-class research

Abstract

Context effects on survey response, caused by the unobserved interaction between beliefs stored in personal memory and triggers generated by the structure of the survey instrument, are a pervasive challenge to survey research. The authors argue that randomized survey experiments on representative samples, when paired with facilitative primes, can enable researchers to model selection into variable context effects, revealing heterogeneity at the population level. The value of the design, and its associated modeling strategy, is demonstrated by its ability to deepen the interpretation of a treatment effect of international competitiveness framing on long-used items drawn from the Phi Delta Kappa/Gallup Poll and the General Social Survey about the quality of schooling in the United States, confidence in the leaders running public education, and support for spending to improve schools.

Keywords

context effect framing effect heterogeneity selection public opinion survey response

Get full access to this article

View all access options for this article.

References

Bushaw

William J.

Lopez

Shane J.

2011. “Betting on Teachers: The 43rd Annual Phi Delta Kappa/Gallup Poll of the Public’s Attitudes toward the Public Schools.” Phi Delta Kappan 93:8–26.

Bushaw

William J.

Lopez

Shane J.

2012. “Public Education in the United States: A Nation Divided. The 44th Annual Phi Delta Kappa/Gallup Poll of the Public’s Attitudes toward the Public Schools.” Phi Delta Kappan 94:9–25.

Chong

Dennis

Druckman

James N.

2007. “Framing Theory.” Annual Review of Political Science 10:103–26.

Chong

Dennis

Druckman

James N.

2011. “Public-elite Interactions: Puzzles in Search of Researchers.” Pp. 170–88 in The Oxford Handbook of American Public Opinion and the Media, edited by Shapiro

R. Y.

Jacobs

L. R.

New York: Oxford University Press.

Druckman

James N.

Fein

Jordan

Leeper

Thomas J.

2012. “A Source of Bias in Public Opinion Stability.” American Political Science Review 106:430–54.

Druckman

James N.

Green

Donald P.

Kuklinski

James H.

Arthur

Lupia

, eds. 2011. Cambridge Handbook of Experimental Political Science. New York: Cambridge University Press.

Druckman

James N.

Kam

Cindy D.

2011. “Students as Experimental Participants: A Defense of the ‘Narrow Data Base.’” Pp. 41–57 in Cambridge Handbook of Experimental Political Science, edited by Druckman

J. N.

D. P.

Green

Kuklinski

J. H.

Lupia

New York: Cambridge University Press.

Druckman

James N.

Kuklinski

James H.

Sigelman

Lee

. 2009. “The Unmet Potential of Interdisciplinary Research: Political Psychological Approaches to Voting and Public Opinion.” Political Behavior 31:485–510.

Elwert

Felix

. 2013. “Graphical Causal Models.” Pp. 245–73 in Handbook of Causal Analysis for Social Research, edited by Morgan

S. L.

Dordrecht, the Netherlands: Springer.

10.

Entman

Robert M.

1993. “Framing: Toward Clarification of a Fractured Paradigm.” Journal of Communication 43:51–58.

11.

Imbens

Guido W.

2004. “Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review.” Review of Economics and Statistics 86:4–29.

12.

Morgan

Stephen L.

Taylor Poppe

Emily S.

2012. “The Consequences of International Comparisons for Public Support of K–12 Education: Evidence from a National Survey Experiment.” Educational Researcher 42:262–68.

13.

Morgan

Stephen L.

Winship

Christopher

. 2015. Counterfactuals and Causal Inference: Methods and Principles for Social Research. 2nd ed. Cambridge, UK: Cambridge University Press.

14.

Mutz

Diana C.

2011. Population-based Survey Experiments. Princeton, NJ: Princeton University Press.

15.

Pearl

Judea

. 2009. Causality: Models, Reasoning, and Inference. 2nd ed. Cambridge, UK: Cambridge University Press.

16.

Pearl

Judea

. 2010. “The Foundations of Causal Inference.” Pp. 75–149 in Sociological Methodology, Vol. 40, edited by Liao

Tim Futing

. Boston, MA: Wiley-Blackwell.

17.

Rasinski

Kenneth A.

1988. “The Effect of Question Wording on Public Support for Government Spending.” GSS Methodological Report No. 54. Chicago, IL: National Opinion Research Center.

18.

Rasinski

Kenneth A.

1989. “The Effect of Question Wording on Public Support for Government Spending.” Public Opinion Quarterly 53:388–94.

19.

Rubin

Donald B.

2005. “Causal Inference Using Potential Outcomes: Design, Modeling, Decisions.” Journal of the American Statistical Association 100:322–31.

20.

Schuman

Howard

. 2008. Method and Meaning in Polls and Surveys. Cambridge, MA: Harvard University Press.

21.

Schuman

Howard

Presser

Stanley

. 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. New York: Academic Press.

22.

Schwarz

Norbert

Seymour

Sudman

, eds. 1992. Context Effects in Social and Psychological Research. New York: Springer-Verlag.

23.

Simon

Adam F.

Davey

Lynn F.

2010. “College Bound: The Effects of Value Frames on Attitudes toward Higher Education Reform.” A Frameworks Research Report. Washington, DC: The Frameworks Institute.

24.

Smith

Tom W.

1987. “That Which We Call Welfare by Any Other Name Would Smell Sweeter: An Analysis of Question Wording on Response Patterns.” Public Opinion Quarterly 51:75–83.

25.

Smith

Tom W.

1991. “Context Effects in the General Social Survey.” Pp. 57–72 in Measurement Errors in Surveys, edited by Biemer

P. P.

Groves

R. M.

Lyberg

L. E.

Mathiowetz

N. A.

Sudman

New York: John Wiley.

26.

Smith

Tom W.

2006. “Wording Effects on the National Spending Priority Items across Time, 1984–2004.” GSS Methodological Report No. 107. Chicago, IL: National Opinion Research Center.

27.

Sniderman

Paul M.

2011. “The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation.” Pp. 102–14 in Cambridge Handbook of Experimental Political Science, edited by Druckman

J. N.

Green

D. P.

Kuklinski

J. H.

Lupia

New York: Cambridge University Press.

28.

Tourangeau

Roger

Rasinski

Kenneth A.

Bradburn

Norman

D’Andrade

Roy

. 1989. “Belief Accessibility and Context Effects in Attitude Measurement.” Journal of Experimental Social Psychology 25:401–21.

29.

Tourangeau

Roger

Rips

Lance J.

Rasinski

Kenneth A.

2000. The Psychology of Survey Response. New York: Cambridge University Press.

30.

Wyer

Robert S.

Srull

Thomas K.

1989. Memory and Cognition in Its Social Context. Hillsdale, NJ: Lawrence Erlbaum.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB