This article offers three contributions for conducting meta-analytic reviews in education research. First, we review publication bias and the challenges it presents for meta-analytic researchers. Second, we review the most recent and optimal techniques for evaluating the presence and influence of publication bias in meta-analyses. We then re-analyze two sets of meta-analytic data from the literacy literature that have been published in different journals. The analyses serve as case examples of the techniques reviewed, and the results demonstrate a range of findings from noticeable instances of publication bias to minimal or no bias. The conclusions have important implications for research, policymaking, and practice. Finally, we discuss recommendations for future research.
American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.
2.
Bangert-DrownsR. L.HurleyM. M.WilkinsonB. (2004). The effects of school-based writing-to-learn interventions on academic achievement: A meta-analysis. Review of Educational Research, 74, 29–58.
3.
BanksG. C.BatchelorJ. H.McDanielM. A. (2010). Smarter people are (a bit) more symmetrical: A meta-analysis of the relationship between intelligence and fluctuating asymmetry. Intelligence, 38, 393–401. doi:10.1016/j.intell.2010.04.00310.1016/j.intell.2010.04.003
4.
BanksG. C.KepesS.McDanielM. A. (2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment20, 182–196. doi: 10.1111/j.1468-2389.2012.00591.x
5.
BanksG. C.McDanielM. A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 40–44. doi:10.1111/j.1754-9434.2010.01292.x10.1111/j.1754-9434.2010.01292.x
6.
BeckerB. J. (2005). The failsafe N or file-drawer number. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 111–126). Chichester, UK: Wiley.
7.
BeggC. B.MazumdarM. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101. doi:1010.2307/25334461010.2307/2533446
8.
BerlinJ. A.GhersiD. (2005). Preventing publication bias: Registries and prospective meta-analysis. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 35–48). Chichester, UK: Wiley.
9.
BorensteinM. (2005). Software for publication bias. In RothsteinH.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 111–126). Chichester, UK: Wiley.
BorensteinM.HedgesL. V.HigginsJ. P.RothsteinH. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley.
12.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
13.
DesjardinsR. (2008). Researching the links between education and well-being. European Journal of Education, 43, 23–35. doi:10.1111/j.1465-3435.2007.00333.x10.1111/j.1465-3435.2007.00333.x
14.
DickersinK. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 11–34). Chichester, UK: Wiley.
15.
DuvalS. J. (2005). The trim and fill method. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 127–144). Chichester, UK: Wiley.
16.
DuvalS. J.TweedieR. L. (2000a). A non- parametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98. doi:10.2307/266952910.2307/2669529
17.
DuvalS. J.TweedieR. L. (2000b). Trim and fill: A simple funnel plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 276–284. doi:210.1111/j.0006-1341X.2000.00455.x210.1111/j.0006-1341X.2000.00455.x
18.
EggerM.SmithG. D.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.
19.
EvangelouE.TrikalinosT. A.IoannidisJ. P. (2005). Unavailability of online supplementary scientific information from articles published in major journals. The FASEB Journal, 19, 1943–1944. doi:1910.1096/fj.1905-4784lsf1910.1096/fj.1905-4784lsf
20.
GreenspanA. (2001, April). The importance of education in today’s economy. Proceedings, pp. 6–11.
21.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
22.
HigginsJ. P.ThompsonS. G.DeeksJ. J.AltmanD. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557–560. doi:510.1136/bmj.1327.7414.1557510.1136/bmj.1327.7414.1557
23.
HopewellS.ClarkeM.MallettS. (2005). Grey literature and systematic reviews. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 48–72). Chichester, UK: Wiley.
24.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
25.
IoannidisJ. P. (1998). Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. Journal of the American Medical Association, 279, 281–286. doi:10.1001/jama.279.4.28110.1001/jama.279.4.281
26.
IoannidisJ. P. (2005). Differentiating biases from genuine heterogeneity: Distinguishing artifactual from substantive effects. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 287–302). Chichester, UK: Wiley.
27.
JickT. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602–611. doi:10.2307/239236610.2307/2392366
28.
JohnstonR.ViaderoD. (2000). Unmet promise: Raising minority achievement. The achievement gap. Education Week, pp. 1–23.
29.
Language a barrier for Latinos in schools. (2010, August5). Education Week.
30.
LauJ.AntmanE. M. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine, 327, 248–254. doi:10.1056/NEJM19920723327040610.1056/NEJM199207233270406
31.
LauJ.SchmidC. H.ChalmersT. C. (1995). Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. Journal of Clinical Epidemiology, 48, 45–57. doi:10.1016/0895-4356(94)00106-Z10.1016/0895-4356(94)00106-Z
32.
McDanielM. A. (2009, April). Cumulative meta- analysis as a publication bias method. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
33.
McDanielM. A.RothsteinH. R.WhetzelD. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927–953. doi:910.1111/j.1744-6570.2006.00059.x910.1111/j.1744-6570.2006.00059.x
34.
MullenB.MuellerleileP.BryantB. (2001). Cumulative meta-analysis: A consideration of indicators of sufficiency and stability. Personality and Social Psychology Bulletin, 27, 1450–1462. doi:1410.1177/014616720127110061410.1177/01461672012711006
35.
National Assessment of Educational Progress. (2009). The nation’s report card: Reading 2009. Washington, DC: Institute of Education Sciences, U.S. Department of Education.
36.
NewmanD. A. (2003). Longitudinal modeling with randomly and systematically missing data: A simulation of ad hoc, maximum likelihood, and multiple imputation techniques. Organizational Research Methods, 6, 328–362. doi:10.1177/109442810325467310.1177/1094428103254673
37.
OrwinR. F. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159. doi:110.3102/10769986008002157110.3102/10769986008002157
38.
PalmerT. M.PetersJ.SuttonA. J.MorenoS. G. (2008). Contour enhanced funnel plots for meta-analysis. The Stata Journal, 8, 242–254.
39.
PearsonP. D.HiebertE. H. (2010). National reports in literacy: Building a scientific base for practice and policy. Educational Researcher, 79, 286–294. doi:210.3102/0013189X10370205210.3102/0013189X10370205
40.
PetersJ.SuttonA. J.JonesD. R.AbramsK. R.RushtonL. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991–996. doi:10.1016/j.jclinepi.2007.11.01010.1016/j.jclinepi.2007.11.010
41.
PiastaS. B.WagnerR. K. (2010). Developing early literacy skills: A meta-analysis of alphabet learning and instruction. Reading Research Quarterly, 45, 8–38. doi:10.1598/RRQ.1545.1591.159210.1598/RRQ.1545.1591.1592
42.
RolstadK.MahoneyK.GlassG. (2005). The big picture: A meta-analysis of program effectiveness research on English language learners. Educational Policy, 19, 572–594. doi:510.1177/0895904805278067510.1177/0895904805278067
43.
RosenthalR. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641. doi:610.1037/0033-2909.1086.1033.1638610.1037/0033-2909.1086.1033.1638
44.
RothsteinH. (2012). Accessing relevant literature. In CooperH. M. (Ed.), APA handbook of research methods in psychology: Vol. 1. Foundations, planning, measures, and psychometrics (pp. 133–144). Washington, DC: American Psychological Association.
45.
RothsteinH. R.SuttonA. J.BorensteinM. (2005a). Publication bias in meta-analyses. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 1–7). Chichester, UK: Wiley.
46.
RothsteinH. R.SuttonA. J.BorensteinM. (2005b). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester, UK: Wiley.
47.
SackettP. R.LarsonJ. R. (1990). Research strategies and tactics in industrial and organizational psychology. In DunnetteM. D.HoughL. M. (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 419–489). Palo Alto, CA: Consulting Psychologists Press.
48.
SchaferJ. L.GrahamJ. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177. doi:10.1037//1082-989X.7.2.14710.1037//1082-989X.7.2.147
49.
SongF.EasterwoodA.GilbodyS.DuleyL.SuttonA. J. (2000). Publication and other selection biases in systematic reviews. Health Technology, 4, 1–115. doi:10.3310/hta410010.3310/hta4100
50.
SongF.ParekhS.HooperL.LokeY. K.RyderJ.SuttonA. J.. . . HarveyI. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14, 1–220. doi:10.3310/hta1408010.3310/hta14080
51.
SterneJ. A.EggerM. (2005). Regression methods to detect publication and other bias in meta-analysis. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 99–110). Chichester, UK: Wiley.
52.
SterneJ. A.GavaghanD.EggerM. (2000). Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119–1129. doi:10.1016/S0895-4356(00)00242-010.1016/S0895-4356(00)00242-0
53.
SterneJ. A.GavaghanD.EggerM. (2005). The funnel plot. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 75–98). Chichester, UK: Wiley.
54.
SterneJ. A.SuttonA. J.IoannidisJ. P.TerrinN.JonesD. R.LauJ.. . . HigginsJ. P. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal, 342, d4002–d4010. doi:10.1136/bmj.d400210.1136/bmj.d4002
55.
SuperfineB. M. (2010). Court-driven reform and equal educational opportunity: Centralization, decentralization, and the shifting judicial role. Review of Educational Research, 80, 108–137. doi:110.3102/0034654309354300110.3102/0034654309354300
56.
SuttonA. J. (2005). Evidence concerning the consequences of publication and related biases. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 175–192). Chichester, UK: Wiley.
57.
SuttonA. J. (2009). Publication bias. In The handbook of research synthesis and meta-analysis (2nd ed., pp. 435–452). New York, NY: Russell Sage Foundation.
58.
SuttonA. J.PigottT. D. (2005). Bias in meta-analysis induced by incompletely reported studies. In RothsteinH.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 223–240). Chichester, UK: Wiley.
59.
TorgersonC. J. (2006). Publication bias: The Achilles’ heel of systematic reviews?British Journal of Educational Studies, 54, 89–102. doi:110.1111/j.1467-8527.2006.00332.x110.1111/j.1467-8527.2006.00332.x
60.
TrikalinosT. A.IoannidisJ. P. (2005). Assessing the evolution of effect sizes over time. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 241–259). Chichester, UK: Wiley.
