Sage Journals: Discover world-class research

Abstract

Outcome-reporting bias occurs when primary studies do not include information about all outcomes measured in a study. When studies omit findings on important measures, efforts to synthesize the research using systematic review techniques will be biased and interpretations of individual studies will be incomplete. Outcome-reporting bias has been well documented in medicine and has been shown to lead to inaccurate assessments of the effects of medical treatments and, in some cases, to omission of reports of harms. This study examines outcome-reporting bias in educational research by comparing the reports of educational interventions from dissertations to their published versions. We find that nonsignificant outcomes were 30% more likely to be omitted from a published study than statistically significant ones.

Keywords

communication experimental research meta-analysis program evaluation research methodology research utilization

Get full access to this article

View all access options for this article.

References

APA Publication and Communication Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839–851. doi:10.1037/0003-066X.63.9.839

Bezeau

Graves

(2001). Statistical power and effect sizes of clinical neuropsychology research. Journal of Clinical Experimental Neuropsychology, 23, 399–406. doi:10.1076/jcen.23.3.399.1181

Chan

A. W.

Altman

D. G.

(2005). Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. British Medical Journal, 330, 753–760. doi:10.1136/bmj.38356.424606.8F

Chan

A. W.

Hróbjartsson

Haahr

M. T.

Gøtzsche

P. C.

Altman

D. G.

(2004). Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association, 291, 2457–2465. doi:10.1001/jama.291.20.2457

Chan

A. W.

Krleza-Jeric

Schmid

Altman

D. G.

(2004). Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Canadian Medical Association Journal, 171, 735–740. doi:10.1503/cmaj.1041086

Cohen

(1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.

Cooper

DeNeve

Charlton

(1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447–452. doi:10.1037//1082-989X.2.4.447

Cooper

Patall

E. A.

(2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological Methods, 14, 165–176. doi:10.1037/a0015565

Francis

D. P.

(in press). How easily can omission of patients, or selection amongst poorly-reproducible measurements, create artificial correlations? Methods for detection and implications for observational research design in cardiology. International Journal of Cardiology.

10.

Greenland

Robins

J. M.

(1985). Estimation of a common effect parameter from sparse follow-up data. Biometrics, 41, 55–68. doi:10.2307/2530643

11.

Hopewell

Loudon

Clarke

M. J.

Oxman

A. D.

Dickersin

(2009). Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews (1), MR000006. doi:10.1002/14651858.MR000006.pub3.

12.

Ioannidis

J. P. A.

(2005). Why most published research findings are false. PLoS Medicine, 2, 696–701. doi:10.1371/journal.pmed.0020124

13.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–32. doi:10.1177/0956797611430953

14.

McCormick

A. C.

Zhao

C. M.

(2005). Rethinking and reframing the Carnegie classification. Change, 37, 50–57. doi:10.3200/CHNG.37.5.51–57

15.

National Science Foundation. (2011). Doctoral recipients from US Universities (NSF13-301). Arlington VA: NSF. http://www.nsf.gov/statistics/sed/2011/start.cfm

16.

Orwin

R. G.

Cordray

D. S.

(1985). Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis. Psychological Bulletin, 97, 134–147. doi:10.1037//0033-2909.97.1.134

17.

Pigott

T. D.

Williams

R. T.

Polanin

J. R.

(2012). Combining individual participant and aggregate data in a meta-analysis with correlational studies. Research Synthesis Methods. Advance online publication. doi:10.1002/jrsm.1051

18.

Robins

Breslow

Greenland

(1986). Estimators of the Mantel–Haenszel variance consistent in both sparse data and large-strata limiting models. Biometrics, 42, 311–323. doi:10.2307/2531052

19.

Rossi

J. S.

(1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology, 58, 646–656.

20.

Rothstein

H. R.

Sutton

A. J.

Borenstein

M. R.

(2005). Publication bias in meta-analysis. Prevention, assessment and adjustments. West Sussex, England: Wiley.

21.

Schulz

K. F.

Altman

D. G.

Moher

, for the CONSORT Group. (2010). CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. British Medical Journal, 340, 332–337. doi:10.1136/bmj.c332

22.

Sedlmeier

Gigerenzer

(1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309–316.

23.

Shadish

W. R.

Haddock

C. K.

(2009). Combining estimates of effect size. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 257–293). New York, NY: Russell Sage Foundation.

24.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632

25.

Spybrook

Raudenbush

S. W.

(2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31, 298–318. doi:10.3102/0162373709339524

26.

Suñé

J. M.

Montoro

J. B.

(2013). Positive outcomes influence the rate and time to publication, but not the impact factor of publications of clinical trial results. PLoS One, 8, 1–8. doi:10.1371/journal.pone.0054583

27.

Turner

E. H.

Matthews

A. M.

Linardatos

Tell

R. A.

Rosenthal

(2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358, 252–260. doi:10.1056/NEJMsa065779

28.

Valentine

J. C.

Pigott

T. D.

Rothstein

H. R.

(2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 35, 215–247. doi:10.3102/1076998609346961

29.

Valentine

J. C.

Thompson

S. G.

(2012). Issues relating to confounding and meta-analysis when including non-randomized studies in systematic reviews on the effects of interventions. Research Synthesis Methods. Advance online publication. doi:10.1002/jrsm.1064

30.

Vedula

S. S.

Bero

Scherer

R. W.

Dickersin

(2009). Outcome reporting in industry-sponsored trials of gabapentin for off-label use. New England Journal of Medicine, 361, 1963–1971. doi:10.1056/NEJMsa0906126