Overviews, or syntheses of research syntheses, have become a popular approach to synthesizing the rapidly expanding body of research and systematic reviews. Despite their popularity, few guidelines exist and the state of the field in education is unclear. The purpose of this study is to describe the prevalence and current state of overviews of education research and to provide further guidance for conducting overviews and advance the evolution of overview methods. A comprehensive search across multiple online databases and gray literature repositories yielded 25 total education–related overviews. Our analysis revealed that many commonly reported aspects of systematic reviews, such as the search, screen, and coding procedures, were regularly unreported. Only a handful of overview authors discussed the synthesis technique and few authors acknowledged the overlap of included systematic reviews. Suggestions and preliminary guidelines for improving the rigor and utility of overviews are provided.
*AndersonR. D. (1983). A consolidation and appraisal of science meta-analyses. Journal of Research in Science Teaching, 20, 497–509. doi:10.1002/tea.3660200511
3.
Apple, Inc. (2014). Filemaker Pro [computer software]. Santa Clara, CA: Author.
4.
BastianH.GlasziouP.ChalmersI. (2010). Seventy-five trials and eleven systematic reviews a day: How will we ever keep up?PLoS Medicine, 7(9), e1000326. doi:10.1371/journal.pmed.1000326
5.
BeckerL. A.OxmanA. D. (2008). Overviews of reviews. In HigginsJ. P. T.GreenS. (Eds.), Cochrane handbook for systematic reviews of interventions: Cochrane book series. Chichester, England: John Wiley.
6.
*BrowneG.GafniA.RobertsJ.ByrneC.MajumdarB. (2004). Effective/efficient mental health programs for school-age children: A synthesis of reviews. Social Science & Medicine, 58, 1367–1384. doi:10.1016/S0277-9536(03)00332-0
*CobbB.LehmannJ.Newman-GoncharR.AlwellM. (2009). Self-determination for students with disabilities: A narrative metasynthesis. Career Development for Exceptional Individuals, 32, 108–114. doi:10.1177/0885728809336654
10.
CombsJ. G.KetchenD. J.Jr.CrookT. R.RothP. L. (2011). Assessing cumulative evidence within “macro” research: Why meta-analysis should be preferred over vote counting. Journal of Management Studies, 48, 178–197. doi:10.1111/j.1467-6486.2009.00899.x
11.
*CookC. R.GreshamF. M.KernL.BarrerasR. B.ThorntonS.CrewsS. D. (2008). Social skills training for secondary students with emotional and/or behavioral disorders: A review and analysis of the meta-analytic literature. Journal of Emotional and Behavioral Disorders, 16, 131–144. doi:10.1177/1063426608314541
12.
CooperH. (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: Sage.
13.
CooperH.HedgesL. V.ValentineJ. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis. Russell Sage Foundation.
14.
CooperH.KoenkaA. C. (2012). The overview of reviews: Unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67, 446–462. doi:10.1037/a0027119
*DiekstraR. F. (2008). Effectiveness of school-based social and emotional education programmes worldwide. In ClouderC.DahlinB.DiekstraR. F. (Eds.), Social and emotional education: An international analysis (pp. 255–312). Santender, Spain: Fundación Marcelino Botin.
17.
*FornessS. R. (2001). Special education and related services: What have we learned from meta-analysis?Exceptionality, 9, 185–197. doi:10.1207/S15327035EX0904_3
18.
*FraserB. J.WalbergH. J.WelchW. W.HattieJ. A. (1987a). Contextual and transactional influences on science outcomes. International Journal of Education Research, 11, 165–186.
19.
*FraserB. J.WalbergH. J.WelchW. W.HattieJ. A. (1987b). Identifying the salient facets of a model of student learning: A synthesis of meta analyses. International Journal of Educational Research, 11, 187–212.
20.
*FraserB. J.WalbergH. J.WelchW. W.HattieJ. A. (1987c). Syntheses of research on factors influencing learning. International Journal of Education Research, 11, 155–164.
21.
GoughD.OliverS.ThomasJ. (2012). An introduction to systematic reviews (1st ed.). London, England: Sage.
22.
*GreenJ.HowesF.WatersE.MaherE.OberklaidF. (2005). Promoting the social and emotional health of primary school-aged children: Reviewing the evidence base for school-based interventions. International Journal of Mental Health Promotion, 7, 30–36. doi:10.1080/14623730.2005.9721872
23.
*GreshamF. M. (1998). Social skills training: Should we raze, remodel, or rebuild?Behavioral Disorders, 24, 19–25.
24.
*GuthrieJ. T.SeifertM.MosbergL. (1983). Research synthesis in reading: Topics, audiences, and citation rates. Reading Research Quarterly, 19, 16–27. doi:10.2307/747334
25.
GuyattG. H.OxmanA. D.VistG. E.KunzR.Falck-YtterY.Alonso-CoelloP.SchünemannH. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336, 924 - 926. doi: 10.1136/bmj.39489.470347.AD
26.
HartlingL.ChisholmA.ThomsonD.DrydenD. M. (2012). A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One, 7(11), e49667. doi:10.1371/journal.pone.0049667
27.
*HattieJ. (1992). Measuring the effects of schooling. Australian Journal of Education, 36(1), 5–13.
28.
*HattieJ. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Oxon, England: Routledge.
29.
HigginsJ. P.GreenS. (Eds.). (2008). Cochrane handbook for systematic reviews of interventions: Cochrane book series. Chichester, England: John Wiley.
IoannidisJ. P. A. (2009). Integration of evidence from multiple meta-analyses: A primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. Canadian Medical Association Journal, 181, 488–493. doi:10.1503/cmaj.081086
32.
KazrinA.DuracJ.AgterosT. (1979). Meta-meta analysis: A new method for evaluating therapy outcome. Behaviour Research and Therapy, 17, 397–399. doi:10.1016/0005-7967(79)90011-1
33.
*KulikJ. A.KulikC. L. C. (1989). The concept of meta-analysis. International Journal of Educational Research, 13, 227–340. doi:10.1016/0883-0355(89)90052-9
34.
LiL.TianJ.TianH.SunR.LiuY.YangK. (2012). Quality and transparency of overviews of systematic reviews. Journal of Evidence-Based Medicine, 5(3), 166-173.
35.
*LipseyM. W.WilsonD. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209. doi:10.1037//0003-066X.48.12.1181
*Lister-SharpD.ChapmanS.Stewart-BrownS.SowdenA. (1999). Health promoting schools and health promotion in schools: Two systematic reviews. Health Technology Assessment, 3, 1–207.
38.
*LloydJ. W.FornessS. R.KavaleK. A. (1998). Some methods are more effective than others. Intervention in School and Clinic, 33, 195–200. doi:10.1177/105345129803300401
39.
*MaagJ. W. (2006). Social skills training for students with emotional and behavioral disorders: A review of reviews. Behavioral Disorders, 32, 4–17.
40.
MerrellK. W.GueldnerB. A.RossS. W.IsavaD. M. (2008). How effective are school bullying intervention programs? A meta-analysis of intervention research. School Psychology Quarterly, 23, 26–42. doi:10.1037/1045-3830.23.1.26
41.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med, 6(6), e1000097. doi:10.1371/journal.pmed1000097
42.
PieperD.AntoineS. L.MorfeldJ. C.MathesT.EikermannM. (2014a). Methodological approaches in conducting overviews: Current state in HTA agencies. Research Synthesis Methods, 5, 187–199. doi:10.1002/jrsm.1107
43.
PieperD.AntoineS. L.NeugebauerE. A.EikermannM. (2014b). Systematic review finds overlapping reviews were not mentioned in every other review. Journal of Clinical Epidemiology, 67, 368–375. doi:10.1016/j.jclinepi.2013.11.007
44.
PieperD.AntoineS. L.NeugebauerE. A.EikermannM. (2014c). Up-to-dateness of reviews is often neglected in overviews: A systematic review. Journal of Clinical Epidemiology, 67, 1302–1308. doi:10.1016/j.jclinepi.2014.08.008
45.
PieperD.BuechterR.JerinicP.EikermannM. (2012). Overviews of reviews often have limited rigor: A systematic review. Journal of Clinical Epidemiology, 65, 1267–1273. doi:10.1016/j.jclinepi.2012.06.015
46.
PigottT. D. (2012). Advances in meta-analysis. New York, NY: Springer.
47.
PolaninJ. R.PigottT. D. (2014). The use of meta-analytic statistical significance testing. Research Synthesis Methods, 6, 63–73. doi:10.1002/jrsm.1124
48.
R Core Team. (2015). R: A language and environment for statistical computing (Version 3.2.3) [Software]. Available from: https://www.r-project.org/
49.
SchmidtF. L.OhI. S. (2013). Methods for second order meta-analysis and illustrative applications. Organizational Behavior and Human Decision Processes, 121, 204–218. doi:10.1016/j.obhdp.2013.03.002
50.
ShadishW. R.LecyJ. D. (2015). The meta-analytic big bang. Research Synthesis Methods. Advanced online publication. doi:10.1002/jrsm.1132
51.
SheaB. J.GrimshawJ. M.WellsG. A.BoersM.AnderssonN.HamelC.… BouterL. M. (2007). Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7, 10. doi:10.1186/1471-2288-7-10
52.
*SipeT. A.CurletteW. L. (1997). A meta-synthesis of factors related to educational achievement: A methodological approach to summarizing and synthesizing meta-analyses. International Journal of Educational Research, 25, 583–698.
53.
SmithV.DevaneD.BegleyC. M.ClarkeM. (2011). Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology, 11, 15. doi:10.1186/1471-2288-11-15
54.
*TamimR. M.BernardR. M.BorokhovskiE.AbramiP. C.SchmidR. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81, 4–28. doi:10.3102/0034654310393361
55.
*TennantR.GoensC.BarlowJ.DayC.Stewart-BrownS. (2007). A systematic review of reviews of interventions to promote mental health and prevent mental health problems in children and young people. Journal of Public Mental Health, 6, 25–32. doi:10.1108/17465729200700005
56.
ThomsonD.FoisyM.OleszczukM.WingertA.ChisholmA.HartlingL. (2013). Overview of reviews in child health: Evidence synthesis and the knowledge base for a specific population. Evidence-Based Child Health: A Cochrane Review Journal, 8, 3–10. doi:10.1002/ebch.1897
57.
ThomsonD.RussellK.BeckerL.KlassenT.HartlingL. (2010). The evolution of a new publication type: Steps and challenges of producing overviews of reviews. Research Synthesis Methods, 1, 198–211. doi:10.1002/jrsm.30
58.
*TorgersonC. J. (2007). The quality of systematic reviews of effectiveness in literacy learning in English: A “tertiary” review. Journal of Research in Reading, 30, 287–315. doi:10.1111/j.1467-9817.2006.00318.x
59.
TtofiM. M.FarringtonD. P. (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56. doi:10.1007/s11292-010-9109-1
60.
ValentineJ. C.CooperH. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13, 130–149. doi:10.1037/10889X.13.2.130
61.
*WangM. C.HaertelG. D.WalbergH. J. (1993). Toward a knowledge base for school learning. Review of Educational Research, 63, 249–294. doi:10.3102/00346543063003249
62.
*WeareK.NindM. (2011). Mental health promotion and problem prevention in schools: What does the evidence say?Health Promotion International, 26, 29–69. doi:10.1093/heapro/dar075
WhitingP.SavovićJ.HigginsJ. P.CaldwellD. M.ReevesB. C.SheaB.… ChurchillR. (2016). ROBIS: a new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225-234.
65.
WilliamsR. T. (2012). Using robust standard errors to combine multiple estimates with meta-analysis (Unpublished doctoral dissertation). Loyola University, Chicago, IL.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
