We conducted a conceptual replication of Pigott et al.’s study of outcome-reporting bias, wherein they compared intervention outcomes reported in unpublished education dissertations with corresponding published versions. For our replication, we identified a sample of 40 special education dissertations with matched journal publications and found that statistically significant intervention outcomes from dissertations were 1.48 times more likely to be published compared with nonsignificant outcomes. Significant moderators of this effect included type of intervention outcome (academic), type of research design (randomized controlled trial), participant race (with samples greater than or equal to 50% non-White), and type of disability/exceptionality (high incidence). We found that few dissertation authors published their work, providing further evidence for the much-needed inclusion of dissertations in systematic reviews.
ChanA. W.HróbjartssonA.HaahrM. T.GøtzscheP. C.AltmanD. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association, 291, 2457–2465. https://doi.org/10.1001/jama.291.20.2457
2.
ChowJ. C.EckholmE. (2018). Do published studies yield larger effect sizes than unpublished studies in education and special education? A meta-review. Educational Psychology Review, 30, 727–744. https://doi.org/10.1007/s10648-018-9437-7
3.
ClarkeB.DoablerC. T.SmolkowskiK.TurturaJ.KostyD.Kurtz-NelsonE.FienH.BakerS. K. (2019). Exploring the relationship between initial mathematics skill and a kindergarten mathematics intervention. Exceptional Children, 85(2), 129–146. https://doi.org/10.1177/0014402918799503
4.
CookB. G.TherrienW. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders, 42(4), 149–158. https://doi.org/10.1177/0198742917709473
5.
DwanK.AltmanD. G.ArnaizJ. A.BloomJ.ChanA.-W.CroninE.DecullierE.EasterbrookP. J.ElmE. V.GambleC.GhersiD.IoannidisJ. P. A.SimesJ.WilliamsonP. R. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLOS ONE, 3(8), Article e3081. https://doi.org/10.1371/journal.pone.0003081
6.
Endrew F. v. Douglas County School District RE-1, 580 U. S. (2017).
FuchsD.FuchsL. S. (2019). On the importance of moderator analysis in intervention research: An introduction to the special issue. Exceptional Children, 85(2), 126–128. https://doi.org/10.1177/0014402918811924
10.
GageN. A.CookB. G.ReichowB. (2017). Publication bias in special education meta-analyses. Exceptional Children, 83(4), 428–445. https://doi.org/10.1177/0014402917691016
11.
GeukeG. G. M.MaricM.MioˇcevicM.WoltersL. H.de HaanE. (2019). Testing mediators of youth intervention outcomes using single-case experimental designs. New Directions for Child and Adolescent Development, 2019(167), 39–64. https://doi.org/10.1002/cad.20310
12.
KuritzS. J.LandisJ. R.KochG. G. (1988). A general overview of Mantel-Haenszel methods: Applications and recent developments. Annual Review of Public Health, 9, 123–160. https://doi.org/10.1146/annurev.pu.09.050188.001011
13.
LyonA. R.ConnorsE.Jensen-DossA.LandesS. J.LewisC. C.McLeodB. D.RuttC.StanickC.WeinerB. J. (2017). Intentional research design in implementation science: Implications for the use of nomothetic and idiographic assessment. TBM, 7, 567–580. https://doi.org/10.1007/s13142-017-0464-6
14.
MaagJ. W.LosinskiM. (2015). Thorny issues and prickly solutions: Publication bias in meta-analytic reviews in the social sciences. Advances in Social Sciences Research Journal, 2(3), 242–253. https://doi.org/10.14738/assrj.23.1044
15.
McLeodB. D.WeiszJ. R. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72(2), 235–251. https://doi.org/10.1037/0022-006X.72.2.235
PigottT. D.ValentineJ. C.PolaninJ. R.WilliamsR. T.CanadaD. D. (2013). Outcome-reporting bias in education research. Educational Researcher, 42(8), 424–432. https://doi.org/10.3102/0013189X13507104
18.
PolaninJ. R.Tanner-SmithE. E.HennessyE. A. (2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067
19.
PreacherK. J.SterbaS. K. (2019). Aptitude-by-treatment interactions in research on educational interventions. Exceptional Children, 85(2), 248–264. https://doi.org/10.1177/0014402918802803
20.
RanganathanP.PrameshC. S.BuyseM. (2016). Common pitfalls in statistical analysis: The perils of multiple testing. Perspectives in Clinical Research, 7(2), 106–107. https://doi.org/10.4103/2229-3485.179436
21.
RothsteinH. R.BushmanB. J. (2012). Publication bias in psychological science: Comment on Ferguson and Brannick (2012). Psychological Methods, 17, 129–136. https://doi.org/10.1037/a0027128
22.
RothsteinH. R.HopewellS. (2009). Grey literature. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (pp. 103–125). Russell Sage Foundation.
23.
RothsteinH. R.SuttonA. J.BorensteinM. (2005). Publication bias in meta-analysis. In RothsteinH. R.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 1–7). John Wiley. https://doi.org/10.1002/0470870168.ch1
24.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage Learning.
25.
TalbottE.De Los ReyesA. (2022). Making sense of multiple data sources: Using single case research design for behavioral decision making. In FarmerT. W.TalbottE.McMasterK. L.LeeD.AcevesT. C. (Eds.), Handbook of Special Education Research: Theory, Methods, and Developmental Processes: Volume 1 (pp. 231–244). Routledge.
26.
TalbottE.MagginD. M.Van AckerE. Y.KummS. (2018). Quality indicators for reviews of research in special education. Exceptionality, 26(4), 245–265. https://doi.org/10.1080/09362835.2017.1283625
27.
TosteJ. R.LoganJ. A. R.ShogrenK. A.BoydB. A. (2023). The next generation of quality indicators for group design research in special education. Exceptional Children, 89(4), 359–378. https://doi.org/10.1177/00144029221150801
28.
TrainorB. P. (2015). Incomplete reporting: Addressing the problem of outcome-reporting bias in educational research [Unpublished doctoral dissertation]. Loyola University Chicago. https://ecommons.luc.edu/luc_diss/1656
29.
VaughnS.RobertsG.CapinP.MiciakJ.ChoE.FletcherJ. M. (2019). How initial word reading and language skills affect reading comprehension outcomes for students with reading difficulties. Exceptional Children, 85(2), 180–196. https://doi.org/10.1177/0014402918782618
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
