In this study, we first develop a framework that presents systematicity as an encompassing orientation toward the application of explicit methods in the practice of literature reviews, informed by the principles of transparency, coverage, saturation, connectedness, universalism, and coherence. We then supplement that conceptual development with empirical insights into the reported practices of systematicity in a sample of 165 published reviews across three journals in organizational research. We finally trace implications for the future conduct of literature reviews, including the potential perils of systematicity without mindfulness.
AdamsR. J.SmartP.HuffA. S. (2017). Shades of grey: Guidelines for working with the grey literature in systematic reviews for management and organizational studies. International Journal of Management Reviews, 19(4), 432–454. https://doi.org/10.1111/ijmr.12102
2.
AguinisH.RamaniR. S.AlabduljaderN. (2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 83–110. https://doi.org/10.5465/annals.2016.0011
3.
AguinisH.ShapiroD. L.AntonacopoulouE. P.CummingsT. G. (2014). Scholarly impact: A pluralist conceptualization. Academy of Management Learning & Education, 13(4), 623–639. https://doi.org/10.5465/amle.2014.0121
4.
AllaK.HallW. D.WhitefordH. A.HeadB. W.MeurkC. S. (2017). How do we define the policy impact of public health research? A systematic review. Health Research and Policy Systems, 15(1), Article 84. https://doi.org/10.1186/s12961-017-0247-z
5.
AlvessonM.SandbergJ. (2020). The problematizing review: A counterpoint to Elsbach and Van Knippenberg’s argument for integrative reviews. Journal of Management Studies, 57(6), 1290–1304. https://doi.org/10.1111/joms.12582
6.
AndersonM. H.LemkenR. K. (2020). Citation context analysis as a method for conducting rigorous and impactful literature reviews. Organizational Research Methods. Advance online publication. https://doi.org/10.1177%2F1094428120969905
7.
AntonsD.BreidbachC.JoshiA.SalgeT. O. (2021). Computational literature reviews: Methods, algorithms, and roadmap. Organizational Research Methods. Advance online publication. https://doi.org/10.1177/1094428121991230
8.
ArkseyH.O’MalleyL. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
9.
BacharachS. B. (1989). Organizational theories: Some criteria for evaluation. The Academy of Management Review, 14(4), 496–515. https://doi.org/10.2307/258555
10.
BartunekJ. M.RynesS. L. (2014). Academics and practitioners are alike and unlike. Journal of Management, 40(5), 1181–1201. https://doi.org/10.1177/0149206314529160
11.
BaumannO.SchmidtJ.StieglitzN. (2019). Effective search in rugged performance landscapes: A review and outlook. Journal of Management, 45(1), 285–318. https://doi.org/10.1177/0149206318808594
BerghD. D.PerryJ.HankeR. (2006). Some predictors of SMJ article impact. Strategic Management Journal, 27(1), 81–100. https://doi.org/10.1002/smj.504
14.
BoothA. (2010). How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. International Journal of Technology Assessment in Health Care, 26(4), 431–435. https://doi.org/10.1017/S0266462310000966
BrinerR. B.DenyerD.RousseauD. M. (2009). Evidence-based management: Concept cleanup time. Academy of Management Perspectives, 23(4), 19–32. https://doi.org/10.5465/amp.23.4.19
18.
BruntonG.OliverS.ThomasJ. (2020). Innovations in framework synthesis as a systematic review method. Research Synthesis Methods, 11, 316–330. https://doi.org/10.1002/jrsm.1399
19.
CardN. (2012). Applied meta-analysis for social science research. Guilford Press.
20.
CarrollC.BoothA.CooperK. (2011). A worked example of “best fit” framework synthesis: A systematic review of views concerning the taking of some potential chemopreventive agents. BMC Medical Research Methodology, 11, Article 29. https://doi.org/10.1186/1471-2288-11-29
21.
CarrollC.RickJ.LeavissJ.FishwickD.BoothA. (2013). A qualitative evidence synthesis of employees’ views of workplace smoking reduction or cessation interventions. BMC Public Health, 13, Article 1095. https://doi.org/10.1186/1471-2458-13-1095
22.
ChiassonM. W. (2015). Avoiding methodological overdose: A declaration for independent ends. Journal of Information Technology, 30(2), 174–176.
23.
CooperH. M.PatallE. A.LindsayJ. J. (2009). Research synthesis and meta-analysis. In BickmanL.RogD. (Eds.), The SAGE handbook of applied social research methods (pp. 344–370). SAGE. https://doi.org/10.4135/9781483348858
24.
CorleyK. G.GioiaD. A. (2011). Building theory about theory building: What constitutes a theoretical contribution. Academy of Management Review, 36(1), 12–32. https://doi.org/10.5465/amr.2009.0486
25.
CroninM. A.GeorgeE. (2020). The why and how of the integrative review. Organizational Research Methods. Advance online publication. https://doi.org/10.1177/1094428120935507
26.
CropanzanoR. (2009). Writing nonempirical articles for Journal of Management: General thoughts and suggestions. Journal of Management, 35(6), 1304–1311. https://doi.org/10.1177/0149206309344118
27.
DalalR. S.BhaveD. P.FisetJ. (2014). Within-person variability in job performance. Journal of Management, 40(5), 1396–1436. https://doi.org/10.1177/0149206314532691
28.
DenyerD.TranfieldD. (2009). Producing a systematic review. In BuchananD. A.BrymanA. (Eds.), The SAGE handbook of organizational research methods (pp. 671–689). SAGE.
29.
DurachC. F.KembroJ.WielandA. (2017). A new paradigm for systematic literature reviews in supply chain management. Journal of Supply Chain Management, 53(4), 67–85. https://doi.org/10.1111/jscm.12145
30.
DuriauV. J.RegerR. K.PfarrerM. D. (2007). A content analysis of the content analysis literature in organization studies: Research themes, data sources, and methodological refinements. Organizational Research Methods, 10(1), 5–34. https://doi.org/10.1177/1094428106289252
31.
EdmondsonA. C.McmanusS. E. (2007). Methodological fit in management field research. Academy of Management Review, 32(4), 1246–1264. https://doi.org/10.5465/amr.2007.26586086
32.
ElsbachK. D.StiglianiI. (2018). Design thinking and organizational culture: A review and framework for future research. Journal of Management, 44(6), 2274–2306. https://doi.org/10.1177/0149206317744252
33.
ErtugG.GruberM.NybergA.SteensmaH. K. (2018). A brief primer on data visualization opportunities in management research. Academy of Management Journal, 61(5), 1613–1625. https://doi.org/10.5465/amj.2018.4005
34.
FelizardoK. R.AnderyG. F.PaulovichF. V.MinghimR.MaldonadoJ. C. (2012). A visual analysis approach to validate the selection review of primary studies in systematic reviews. Information and Software Technology, 54(10), 1079–1091. https://doi.org/10.1016/j.infsof.2012.04.003
35.
FelizardoK. R.SallehN.MartinsR. M.MendesE.MacDonellS. G.MaldonadoJ. C. (2011). Using visual text mining to support the study selection activity in systematic literature reviews. In 2011 International Symposium on Empirical Software Engineering and Measurement (pp. 77–86). IEEE. https://dx.doi.org/10.1109/esem.2011.16
36.
FengL.ChiamY. K.LoS. K. (2017). Text-mining techniques and tools for systematic literature reviews: A systematic literature review. In 2017 24th Asia-Pacific Software Engineering Conference (APSEC) (pp. 41–50). IEEE. https://dx.doi.org/10.1109/apsec.2017.10
37.
GaurA.KumarM. (2018). A systematic approach to conducting review studies: An assessment of content analysis in 25 years of IB research. Journal of World Business, 53(2), 280–289. https://doi.org/10.1016/j.jwb.2017.11.003
GlaserB.StraussA. (1967). The discovery of grounded theory: Strategies for qualitative research. Sociology Press.
40.
GrantM. J.BoothA. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
41.
GreenhalghT.RobertG.MacfarlaneF.BateP.KyriakidouO.PeacockR. (2005). Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Social Science & Medicine, 61(2), 417–430. https://doi.org/10.1016/j.socscimed.2004.12.001
42.
GreenhalghT.ThorneS.MalterudK. (2018). Time to challenge the spurious hierarchy of systematic over narrative reviews. European Journal of Clinical Investigation, 48(6), Article e12931. https://doi.org/10.1111/eci.12931
43.
HaddawayN. R.WoodcockP.MacuraB.CollinsA. (2015). Making literature reviews more reliable through application of lessons from systematic reviews. Conservation Biology, 29(6), 1596–1605. https://doi.org/10.1111/cobi.12541
HieblM. R. W. (2021). Sample selection in systematic literature reviews of management research. Organizational Research Methods. Advance online publication. https://doi.org/10.1177/10944281211008652
46.
HillmanA. J.WithersM. C.CollinsB. J. (2009). Resource dependence theory: A review. Journal of Management, 35(6), 1404–1427. https://doi.org/10.1177/0149206309343469
Hoyningen-HueneP. (2013). Systematicity: The nature of science. Oxford University Press.
49.
JudgeT. A.CableD. M.ColbertA. E.RynesS. L. (2007). What causes a management article to be cited—Article, author, or journal. Academy of Management Journal, 50(3), 491–506. https://doi.org/10.5465/amj.2007.25525577
50.
KastnerM.StrausS. E.McKibbonK. A.GoldsmithC. H. (2009). The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. Journal of Clinical Epidemiology, 62(2), 149–157. https://doi.org/10.1016/j.jclinepi.2008.06.001
51.
KastnerM.TriccoA. C.SoobiahC.LillieE.PerrierL.HorsleyT.WelchV.CogoE.AntonyJ.StrausS. E. (2012). What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Medical Research Methodology, 12, Article 114. https://doi.org/10.1186/1471-2288-12-114
KnaflK.WhittemoreR. (2017). Top 10 tips for undertaking synthesis research. Research in Nursing & Health, 40(3), 189–193. https://doi.org/10.1002/nur.21790
55.
KobayashiV. B.MolS. T.BerkersH. A.KismihókG.Den HartogD. N. (2018). Text mining in organizational research. Organizational Research Methods, 21(3), 733–765. https://doi.org/10.1177/1094428117722619
56.
KunischS.MenzM.BartunekJ. M.CardinalL. B.DenyerD. (2018). Feature Topic at Organizational Research Methods. Organizational Research Methods, 21(3), 519–523. https://doi.org/10.1177/1094428118770750
57.
LaplumeA. O.SonparK.LitzR. A. (2008). Stakeholder theory: Reviewing a theory that moves us. Journal of Management, 34(6), 1152–1189. https://doi.org/10.1177/0149206308324322
58.
LeidnerD. (2018). Review and theory symbiosis: An introspective retrospective. Journal of the Association for Information Systems, 19(06), 552–567. https://doi.org/10.17705/1jais.00501
59.
LiberatiA.AltmanD. G.TetzlaffJ.MulrowC.GotzscheP. C.IoannidisJ. P.ClarkeM.DevereauxP. J.KleijnenJ.MoherD. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med, 6(7), Article e1000100. https://doi.org/10.1371/journal.pmed.1000100
60.
LunnyC.BrennanS. E.McDonaldS.MvKenzieJ. E. (2017). Toward a comprehensive evidence map of overview of systematic review methods. Systematic Reviews, 6(231), 1–27. https://doi.org/10.1186/s13643-017-0617-1
61.
MaddenA.BaileyC.AlfesK.FletcherL. (2018). Using narrative evidence synthesis in HRM research: An overview of the method, its application, and the lessons learned. Human Resource Management, 57(2), 641–657. https://doi.org/10.1002/hrm.21858.
62.
MacLureM. (2005). ‘Clarity bordering on stupidity’: Where’s the quality in systematic review?Journal of Education Policy, 20(4). 393–416.
63.
MertonR. K. (1973). The normative structure of science. In StorerN. M. (Ed.), The sociology of science: Theoretical and empirical investigations (pp. 223–280). The University of Chicago Press. (Original work published 1942)
64.
ParéG.TateM.JohnstoneD.KitsiouS. (2016). Contextualizing the twin concepts of systematicity and transparency in information systems literature reviews. European Journal of Information Systems, 25(6), 493–508. https://doi.org/10.1057/s41303-016-0020-3
65.
ParéG.TrudelM.-C.JaanaM.KitsiouS. (2015). Synthesizing information systems knowledge: A typology of literature reviews. Information & Management, 52(2), 183–199. https://doi.org/10.1016/j.im.2014.08.008
66.
PolonioliA. (2020). In search of better science: On the epistemic costs of systematic reviews and the need for a pluralistic stance to literature search. Scientometrics, 122(2), 1267–1274. https://doi.org/10.1007/s11192-019-03333-3
67.
PostC.SaralaR.GatrellC.PrescottJ. E. (2020). Advancing theory with review articles. Journal of Management Studies, 57(2), 351–376. https://doi.org/10.1111/joms.12549
68.
PrattM. G.SonensheinS.FeldmanM. S. (2020). Moving beyond templates: A bricolage approach to conducting trustworthy qualitative research. Organizational Research Methods. Advance online publication. https://doi.org/10.1177/1094428120927466
69.
PullinA. S.StewartG. B. (2006). Guidelines for systematic review in conservation and environmental management. Conservation Biology, 20(6), 1647–1656. https://doi.org/10.1111/j.1523-1739.2006.00485.x
70.
RousseauD. M.ManningJ.DenyerD. (2008). Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. Academy of Management Annals, 2(1), 475–515. https://doi.org/10.5465/19416520802211651
71.
RoweF. (2014). What literature review is not: Diversity, boundaries and recommendations. European Journal of Information Systems, 23(3), 241–255. https://doi.org/10.1057/ejis.2014.7
72.
SamnaniS. S.VaskaM.AhmedS.TurinT. C. (2017). Review typology: The basic types of reviews for synthesizing evidence for the purpose of knowledge translation. Journal of the College of Physicians and Surgeons Pakistan, 27(10), 635–641.
73.
SandbergJ.AlvessonM. (2021). Meanings of theory: Clarifying theory through typification. Journal of Management Studies, 58(2), 487–516. https://doi.org/10.1111/joms.12587
74.
SandelowskiM.BarrosoJ. (2003). Classifying the findings in qualitative studies. Qualitative Health Research, 13(7), 905–923. https://doi.org/10.1177/1049732303253488
75.
SandelowskiM.VoilsC. I.BarrosoJ. (2006). Defining and distinguishing mixed research synthesis studies. Research in the Schools, 13(1), 29–40.
76.
ShortJ. C.PalmerT. B. (2008). The application of DICTION to content analysis research in strategic management. Organizational Research Methods, 11(4), 727–752. https://doi.org/10.1177/1094428107304534
77.
SiddawayA. P.WoodA. M.HedgesL. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70(1), 747–770. https://doi.org/10.1146/annurev-psych-010418-102803
78.
SimsekZ.FoxB. C.HeaveyC. (2015). “What’s past is prologue”: A framework, review and future directions for organizational research on imprinting. Journal of Management, 41(1), 288–317. https://doi.org/10.1177/0149206314553276
79.
SuriH.ClarkeD. (2009). Advancements in research synthesis methods: From a methodologically inclusive perspective. Review of Educational Research, 79(1), 395–430. https://doi.org/10.3102/0034654308326349
80.
TangC.ZhuY.BlackleyS. V.PlasekJ. M.WanM.ZhouL.MaJ.BatesD. W. (2019). Visualizing literature review theme evolution on timeline maps: Comparison across disciplines. IEEE Access, 7, 90597–90607. https://doi.org/10.1109/access.2019.2925706
81.
TranfieldD.DenyerD.SmartP. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222. https://doi.org/10.1111/1467-8551.00375
82.
TriccoA. C.SoobiahC.AntonyJ.CogoE.MacDonaldH.LillieE.TranJ.D’SouzaJ.HuiW.PerrierL.WelchV.HorsleyT.StrausS. E.KastnerM. (2016). A scoping review identifies multiple emerging knowledge synthesis methods, but few studies operationalize the method. Journal of Clinical Epidemiology, 73, 19–28. https://doi.org/10.1016/j.jclinepi.2015.08.030
83.
TufteE. R. (2001). The visual display of quantitative information. Graphics Press.
84.
TurnerJ. R.BakerR.KellnerF. (2018). Theoretical literature review: Tracing the life cycle of a theory and its verified and falsified statements. Human Resource Development Review, 17(1), 34–61. https://doi.org/10.1177/1534484317749680
85.
van EckN. J.WaltmanL. (2014). Visualizing bibliometric networks. In DingY.RousseauR.WolframD. (Eds.), Measuring scholarly impact (pp. 285–320). Springer International Publishing. https://doi.org/10.1007/978-3-319-10377-8_13
86.
XiaoY.WatsonM. (2019). Guidance on conducting a systematic literature review. Journal of Planning Education and Research, 39(1), 93–112. https://doi.org/10.1177/0739456x17723971
87.
ZupicI.ČaterT. (2015). Bibliometric methods in management and organization. Organizational Research Methods, 18(3), 429–472. https://doi.org/10.1177/1094428114562629
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
