The substantial volume, continued growth, and resulting complexity of the scientific literature not only increases the need for systematic, replicable, and rigorous literature reviews, but also highlights the natural limits of human researchers’ information processing capabilities. In search of a solution to this dilemma, computational techniques are beginning to support human researchers in synthesizing large bodies of literature. However, actionable methodological guidance on how to design, conduct, and document such computationally augmented literature reviews is lacking to date. We respond by introducing and defining computational literature reviews (CLRs) as a new review method and put forward a six-step roadmap, covering the CLR process from identifying the review objectives to selecting algorithms and reporting findings. We make the CLR method accessible to novice and expert users alike by identifying critical design decisions and typical challenges for each step and provide practical guidelines for tailoring the CLR method to four conceptual review goals. As such, we present CLRs as a literature review method where the choice, design, and implementation of a CLR are guided by specific review objectives, methodological capabilities, and resource constraints of the human researcher.
AdevaJ. G.AtxaJ. P.CarrilloM. U.ZengotitabengoaE. A. (2014). Automatic text classification to support systematic reviews in medicine. Expert Systems with Applications, 41(4), 1498–1508.
2.
AntonsD.BreidbachC. F. (2018). Big data, big insights? Advancing service innovation and design with machine learning. Journal of Service Research, 21(1), 17–39.
3.
AntonsD.GrünwaldE.CichyP.SalgeT. O. (2020). The application of text mining methods in innovation research: Current state, evolution patterns, and development priorities. R&D Management, 50, 329–351.
4.
AntonsD.JoshiA. M.SalgeT. O. (2019). Content, contribution, and knowledge consumption: Uncovering hidden topic structure and rhetorical signals in scientific texts. Journal of Management, 45(7), 3035–3076.
5.
AntonsD.KleerR.SalgeT. O. (2016). Mapping the topic landscape of JPIM, 1984–2013: In search of hidden structures and development trajectories. Journal of Product Innovation Management, 33(6), 726–749.
6.
AsgariE.BastaniK. (2017). The utility of hierarchical Dirichlet process for relationship detection of latent constructs. Academy of Management Proceedings, 2017(1), 16300.
7.
BadgerD.NurstenJ.WilliamsP.WoodwardM. (2000). Should all literature reviews be systematic?Evaluation & Research in Education, 14(3-4), 220–230.
8.
BleiD. M. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77–84.
9.
BleiD. M.NgA. Y.JordanM. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
10.
BleiD. M.LaffertyJ. D. (2007). A correlated topic model of science. The Annals of Applied Statistics, 1(1), 17–35.
11.
Boyd-GraberJ.HuY.MimnoD. (2017). Applications of topic models. Foundations and Trends in Information Retrieval, 11(2–3), 143–296.
12.
BrinerR. B.DenyerD.RousseauD. M. (2009). Evidence-based management: Concept cleanup time?Academy of Management Perspectives, 23(4), 19–32.
13.
BrussM.AlbersM. J.McNamaraD. (2004). Changes in scientific articles over two hundred years: A coh-metrix analysis. In SIGDOC ‘04: Proceedings of the 22nd Annual International Conference on Design of Communication: The Engineering of Quality Documentation (pp. 104–109). ACM.
14.
BrustL.BreidbachC. F.AntonsD.SalgeT. O. (2017). Service-dominant logic and information systems research: A review and analysis using topic modeling. In ICIS 2017 Proceedings. AIS.
15.
BurrellG.MorganG. (1979). Sociological paradigms and organizational analysis. Heinemann.
16.
CalińskiT.HarabaszJ. (1974). A dendrite method for cluster analysis. Communications in Statistics—Theory and Methods, 3(1), 1–27.
17.
CambriaE.WhiteB. (2014). Jumping NLP curves: A review of natural language processing research. IEEE Computational Intelligence Magazine, 9(2), 48–57.
18.
ChaeB.OlsonD. (2018). A topical exploration of the intellectual development of decision sciences 1975–2016. Decision Sciences. https://doi.org/10.1111/deci.12326.
19.
ChangJ.GerrishS.WangC.Boyd-GraberJ. L.BleiD. M. (2009). Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems (pp. 288–296). NeurIPS.
20.
ChenK.ZhangY.FuX. (2019). International research collaboration: An emerging domain of innovation studies?Research Policy, 48(1), 149–168.
21.
CooleyM. (1996). On human-machine symbiosis. In GillK. S. (Ed.), Human machine symbiosis (pp. 69–100). Springer.
22.
CorleyK. G.GioiaD. A. (2011). Building theory about theory building: What constitutes a theoretical contribution?Academy of Management Review, 36(1), 12–32.
23.
CretchleyJ.RooneyD.GalloisC. (2010). Mapping a 40-year history with Leximancer: Themes and concepts in the Journal of Cross-Cultural Psychology. Journal of Cross-Cultural Psychology, 41(3), 318–328.
24.
CropanzanoR. (2009). Writing nonempirical articles for Journal of Management: General thoughts and suggestions. Sage.
25.
DahlanderL.GannD. M. (2010). How open is innovation?Research Policy, 39(6), 699–709.
26.
DavisJ. P.EisenhardtK. M.BinghamC. B. (2007). Developing theory through simulation methods. Academy of Management Review, 32(2), 480–499.
27.
DebortoliS.MüllerO.JunglasI.vom BrockeJ. (2016). Text mining for information systems researchers: An annotated topic modeling tutorial. Communications of the Association for Information Systems, 39(1), 7.
28.
DeerwesterS.DumaisS. T.FurnasG. W.LandauerT. K.HarshmanR. (1990). Indexing by latent semantic analysis. Journal of the American society for information science, 41(6), 391–407.
29.
DenyerD.TranfieldD. (2009). Producing a systematic review In BuchananD. A.BrymanA. (Eds.), The SAGE handbook of organizational research methods (pp. 671–689). Sage.
30.
DoanA.RamakrishnanR.VaithyanathanS. (2006). Managing information extraction: State of the art and research directions. Paper presented at the ACM SIGMOD International Conference on Management of Data.
31.
EomS. B. (2009). Author cocitation analysis: Quantitative methods for mapping the intellectual structure of an academic discipline. Information Science Reference.
32.
FeldmanR.SangerJ. (2007). The text mining handbook: Advanced approaches in analyzing unstructured data. Cambridge University Press.
33.
FisherG.AguinisH. (2017). Using theory elaboration to make theoretical advancements. Organizational Research Methods, 20(3), 438–464.
FurukawaT.MoriK.ArinoK.HayashiK.ShirakawaN. (2015). Identifying the evolutionary process of emerging technologies: A chronological network analysis of World Wide Web conference sessions. Technological Forecasting and Social Change, 91, 280–294.
36.
GassonS. (2003). Human-centered vs. user-centered approaches to information system design. Journal of Information Technology Theory and Application, 5(2), 5.
37.
GatrellC.BreslinD. (2017). Editors’ statement. International Journal of Management Reviews, 19(1), 1–3.
38.
GillK. S. (1990). Summary of human-centered systems research in Europe. University of Brighton, SEAKE Centre.
39.
GraesserA. C.McNamaraD. S.CaiZ.ConleyM.LiH.PennebakerJ. (2014). Coh-Metrix measures text characteristics at multiple levels of language and discourse. Elementary School Journal, 115(2), 210–229.
40.
GriffinA.SomermeyerS. (Eds.). (2007). The PDMA toolbook 3 for new product development (Vol. 3). John Wiley.
41.
GriffithsT. L.SteyversM. (2004). Finding scientific topics. Proceedings of the National Academy of Sciences, 101(Suppl. 1), 5228–5235.
42.
GrimmerJ.KingG. (2011). General purpose computer-assisted clustering and conceptualization. Proceedings of the National Academy of Sciences, 108(7), 2643–2650.
43.
GrimmerJ.StewartB. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 267–297.
44.
HanniganT. R.HaansR. F.VakiliK.TchalianH.GlaserV. L.WangM. S.KaplanS.JenningsP. D. (2019). Topic modeling in management research: Rendering new theory from textual data. Academy of Management Annals, 13(2), 586–632.
45.
HeQ.ChenB.PeiJ.QiuB.MitraP.GilesL. (2009). Detecting topic evolution in scientific literature: How can citations help?Paper presented at the 18th ACM Conference on Information and Knowledge Management.
HöllererM. A.JancsaryD.BarberioV.MeyerR. E. (2020). The interlinking theorization of management concepts: Cohesion and semantic equivalence in management knowledge. Organization Studies. 41, 1284–1310.
48.
HoppC.AntonsD.KaminskiJ.SalgeT. O. (2018). The topic landscape of disruption research—A call for consolidation, reconciliation, and generalization. Journal of Product Innovation Management, 35(3), 458–487.
49.
HornikK.GrünB. (2011). topicmodels: An R package for fitting topic models. Journal of Statistical Software, 40(13), 1–30.
50.
IttipanuvatV.FujitaK.SakataI.KajikawaY. (2014). Finding linkage between technology and social issue: A literature based discovery approach. Journal of Engineering and Technology Management, 32, 160–184.
51.
JinhaA. E. (2010). Article 50 million: An estimate of the number of scholarly articles in existence. Learned Publishing, 23(3), 258–263.
52.
KuncM.MortensonM. J.VidgenR. (2018). A computational literature review of the field of system dynamics from 1974 to 2017. Journal of Simulation, 12(2), 115–127.
53.
LeeH.KangP. (2018). Identifying core topics in technology and innovation management studies: A topic model approach. Journal of Technology Transfer, 43(5), 1291–1317.
54.
Leximancer. (2018). Leximancer User Guide: Release 4.5.
55.
LloretE.PalomarM. (2012). Text summarisation in progress: A literature review. Artificial Intelligence Review, 37(1), 1–41.
56.
MarchJ. G. (1991). Exploration and exploitation in organizational learning. Organization Science, 2(1), 71–87.
57.
MacInnisD. J. (2011). A framework for conceptual contributions in marketing. Journal of Marketing, 75(4), 136–154.
58.
MarkowitzD. M.HancockJ. T. (2016). Linguistic obfuscation in fraudulent science. Journal of Language and Social Psychology, 35(4), 435–445.
59.
McCarthyP. M.LehenbauerB. M.HallC.DuranN. D.FujiwaraY.McNamaraD. S. (2007). A Coh-Metrix analysis of discourse variation in the texts of Japanese, American, and British scientists. Foreign Languages for Specific Purposes, 6, 46–77.
60.
McCarthyP. M.LewisG. A.DuftyD. F.McNamaraD. S. (2006). Analyzing writing styles with Coh-Metrix. Paper presented at the FLAIRS Conference.
61.
McKennyA. F.AguinisH.ShortJ. C.AnglinA. H. (2018). What doesn’t get measured does exist: Improving the accuracy of computer-aided text analysis. Journal of Management, 44(7), 2909–2933.
62.
McKinleyW.MoneM. A.MoonG. (1999). Determinants and development of schools in organization theory. Academy of Management Review, 24(4), 634–648.
63.
McNamaraD. S.GraesserA. C. (2012). Coh-Metrix: An automated tool for theoretical and applied natural language processing. In McCarthyP. M.Boonthum-DeneckeC. (Eds.), Applied natural language processing: Identification, investigation and resolution (pp. 188–205): IGI Global.
64.
McNamaraD. S.GraesserA. C.McCarthyP. M.CaiZ. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
65.
MilesM. B.HubermanA. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.
66.
MortensonM. J.VidgenR. (2016). A computational literature review of the technology acceptance model. International Journal of Information Management, 36(6), 1248–1259.
67.
MulrowC. D. (1994). Systematic reviews: Rationale for systematic reviews. BMJ, 309(6954), 597–599.
68.
NeelyJ. G.MagitA. E.RichJ. T.VoelkerC. C.WangE. W.PanielloR. C.…BradleyJ. P. (2010). A practical guide to understanding systematic reviews and meta-analyses. Otolaryngology—Head and Neck Surgery, 142(1), 6–14.
69.
NerurS. P.RasheedA. A.NatarajanV. (2008). The intellectual structure of the strategic management field: An author co-citation analysis. Strategic Management Journal, 29(3), 319–336.
70.
NosellaA.CantarelloS.FilippiniR. (2012). The intellectual structure of organizational ambidexterity: A bibliographic investigation into the state of the art. Strategic Organization, 10(4), 450–465.
71.
OdijkD.BurscherB.VliegenthartR.De RijkeM. (2013). Automatic thematic content analysis: Finding frames in news. Paper presented at the International Conference on Social Informatics.
72.
ParéG.TateM.JohnstoneD.KitsiouS. (2016). Contextualizing the twin concepts of systematicity and transparency in information systems literature reviews. European Journal of Information Systems, 25(6), 493–508.
73.
PaulM.GirjuR. (2009). Topic modeling of research fields: An interdisciplinary perspective. Paper presented at the Proceedings of the International Conference RANLP-2009.
PietschA.-S.LessmannS. (2018). Topic modeling for analyzing open-ended survey responses. Journal of Business Analytics, 1(2), 93–116.
76.
PostC.SaralaR.GatrellC.PrescottJ. E. (2020). Advancing theory with review articles. Journal of Management Studies, 57(2), 351–376.
77.
RaischS.KrakowskiS. (2020). Artificial intelligence and management: The automation-augmentation paradox. Academy of Management Review. https://doi.org/10.5465/2018.0072
78.
RandhawaK.WildenR.HohbergerJ. (2016). A bibliometric review of open innovation: Setting a research agenda. Journal of Product Innovation Management, 33(6), 750–772.
79.
RosenthalR.DiMatteoM. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52(1), 59–82.
80.
RüdigerM.AntonsD.SalgeT. O. (2017). From text to data: On the role and effect of text pre-processing in text mining research. Paper presented at the Academy of Management Proceedings.
81.
SchmiedelT.MüllerO.vom BrockeJ. (2019). Topic modeling as a strategy of inquiry in organizational research: A tutorial with an application example on organizational culture. Organizational Research Methods, 22, 941–968.
82.
ShortJ. (2009). The art of writing a review article. Journal of Management, 35(6), 1312–1317.
83.
ShortJ. C.BrobergJ. C.CogliserC. C.BrighamK. H. (2010). Construct validation using computer-aided text analysis (CATA): An illustration using entrepreneurial orientation. Organizational Research Methods, 13(2), 320–347.
84.
SiddawayA. P.WoodA. M.HedgesL. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70, 747–770.
85.
SimonH. A. (1990). Invariants of human behavior. Annual Review of Psychology, 41(1), 1–20.
86.
SteyversM.SmythP.Rosen-ZviM.GriffithsT. (2004). Probabilistic author-topic models for information discovery. Paper presented at the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
87.
SuddabyR. (2010). Editor’s comments: Construct clarity in theories. Academy of Management Review, 35(3), 346–357.
88.
TangM.-C.ChengY. J.ChenK. H. (2017). A longitudinal study of intellectual cohesion in digital humanities using bibliometric analyses. Scientometrics, 113(2), 985–1008.
89.
TanskanenK.AholaT.AminoffA.BraggeJ.KaipiaR.KauppiK. (2017). Towards evidence-based management of external resources: Developing design propositions and future research avenues through research synthesis. Research Policy, 46(6), 1087–1105.
90.
TehY. W.JordanM. I.BealM. J.BleiD. M. (2006). Sharing clusters among related groups: Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476), 1566–1581.
91.
TorracoR. J. (2016). Writing integrative literature reviews: Using the past and present to explore the future. Human Resource Development Review, 15(4), 404–428.
92.
TuanA.DalliD.GandolfoA.GravinaA. (2019). Theories and methods in CSRC research: A systematic literature review. Corporate Communications: An International Journal, 24(2), 212–231.
93.
UpretiB. R.AsatianiA.MaloP. (2016). To reach the clouds: Application of topic models to the meta-review on cloud computing literature. Paper presented at the 2016 49th Hawaii International Conference on System Sciences.
94.
VogelR.GüttelW. H. (2013). The dynamic capability view in strategic management: A bibliometric review. International Journal of Management Reviews, 15(4), 426–446.
95.
VogelR.MasalD. (2015). Public leadership: A review of the literature and framework for future research. Public Management Review, 17(8), 1165–1189.
96.
WangX.McCallumA.WeiX. (2007). Topical n-grams: Phrase and topic discovery, with an application to information retrieval. In Seventh IEEE International Conference on Data Mining (ICDM 2007) (pp. 697–702). IEEE.
WebsterJ.WatsonR. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), xiii–xxiii.
99.
WildenR.AkakaM. A.KarpenI. O.HohbergerJ. (2017). The evolution and prospects of service-dominant logic: An investigation of past, present, and future research. Journal of Service Research, 20(4), 345–361.
100.
WildenR.HohbergerJ.DevinneyT. M.LavieD. (2018). Revisiting James March (1991): Whither exploration and exploitation?Strategic Organization, 16, 352–369.
101.
WriterB. (2019). Lithium-ion batteries. Springer.
102.
XuW.LiuX.GongY. (2003). Document clustering based on non-negative matrix factorization. Paper presented at the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
103.
YanE.DingY. (2012). Scholarly network similarities: How bibliographic coupling networks, citation networks, cocitation networks, topical networks, coauthorship networks, and coword networks relate to each other. Journal of the American Society for Information Science and Technology, 63(7), 1313–1326.
104.
ZhangJ.SongY.ZhangC.LiuS. (2010). Evolutionary hierarchical Dirichlet processes for multiple correlated time-varying corpora. Paper presented at the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
105.
ZupicI.CaterT. (2013). Bibliometric methods in management and organization: A review. Paper presented at the Academy of Management Proceedings.
