Sage Journals: Discover world-class research

Abstract

The Delphi method has been used as a way to reach consensus among experts established in the 1950s. The method was originally conceived to be used as a forecasting instrument for business in a mixed-method study. The Delphi method is usually conducted with elements of anonymity, iteration, controlled feedback and statistical group response. Delphi method had traditionally used expert opinions through methods such as interviews and brainstorming for the initial point of studies conducted. However, with the expanding volumes of research papers and articles in this age, can techniques such as Traditional Literature Review (TLR) and Systematic Literature Review (SLR) also be viable technique to be an initial point of studies for a Delphi study. As such this study aims 1) to examine and discover whether TLR and SLR are appropriate techniques to be used in a Delphi study and 2) construct a Delphi study process overview. This study adopts a methodology in which seven articles that have used TLR and SLR in their Delphi study will be analyzed. The results shows that TLR and SLR are appropriate techniques to be used in a Delphi study and that there are two types of processes to incorporate them into the study. We have also constructed a Delphi process overview through our analysis. We hope that the results of this study will be able help researchers who are interested in doing a Delphi study to know on what are the benefits of TLR and SLR in these studies along with how to incorporate them.

Keywords

delphi method delphi technique systematic literature review traditional literature review

Introduction

The Delphi Method is a technique first put-forward in the 1950s Schmalz et al. (2020) as a way to reach a consensus among experts through anonymous feedback among themselves in repeated rounds (Nasa et al., 2021). As this method was originally conceived to be used as a forecasting instrument for business and other sectors through collective judgement Nasa, Jain, and Juneja (2021) it was called “Delphi”. The “Delphi” term had initially started in ancient Greek as another name for the precinct of “Pytho” which was the seat “Pythia”. “Phythia” was a renowned oracle who was involved in decision-making of important rulings. As such it can be inferred that the Delphi method was constructed as way to predict issues and make decision based on predictions found.

The Delphi method is usually conducted with elements of anonymity, iteration, controlled feedback, and statistical group response as explained in Table 1 by (Smarandache et al., 2020; von der Gracht, 2012).

Table 1.

Delphi Method Elements.

Element	Explanation
Anonymity	A characteristic of the delphi studies are the participants, whom are the interviewees in the study does not know who the other interviewees in the study as it will organized by the moderator and facilitator. The moderator and facilitator of this study will be the researcher. This is as the researcher will be the one to interview the interviewees as well as analyze the responses of the interviewees
Iteration	The delphi method procedure is carried out in a series of rounds. This is to find stability and if possible, a consensus from the interviewees, this is as the purpose of iteration is to “reduce intentional or unintentional noise, such as irrelevant, non-productive and potentially frustrating communication” (von der Gracht, 2012). This study will be iterated until stability is found as the when there is stability in the responses from the interviewees, it can be said that the responses from the interviewees are finalized.
Controlled feedback	This is as the facilitator decides the type of feedback and its provision in which after each round the feedback data will be analyzed and re-stated in aggregated form. In this study, the researcher will be doing this as they are facilitator.
Statistical group response	This are measures such as central tendency (mode, median, mean), dispersion (standard deviation, interquartile ranges, and etc), and frequency distributions that can presented both numerically and/or graphically. The responses will be given to each interviewee after each round as to allow to decide whether to change their previous answer or stand firm with initial decision. This to measure whether the study has gotten either or both the stability and consensus in a study and that the convergence of opinions are achieved.

Traditional Literature Review (TLR) has been stated to be a retrospective account of previous research on certain topic (Li & Wang, 2018). Meanwhile, Systematic Literature Review (SLR) has been stated as a means of evaluating and interpreting all available research significant to a singular research question, topic area, or phenomenon of interest (Kitchenham et al., 2016). TLR and SLR has been said to be important techniques in a study as they are able to contribute to research by allowing a researcher to understand the terms and concepts in a research topic or area (Kitchenham et al., 2016; Li & Wang, 2018).

With the expanding volume of research papers and articles in this age encompassing all study areas, these articles can contribute to multiple types of studies. As such does this include Delphi studies which had traditionally used expert opinions through methods such as interviews and brainstorming for the initial point of studies conducted.

The goal of this study is to find out whether TLR and SLR are appropriate techniques to be used in a Delphi study and to construct an overview of a Delphi study. Moving forward, the discussion of this study adopted methodology will be presented. Followed by, discussion of Delphi studies which had used TLR or SLR in their research. Then, the discussion regarding this study will be elaborated. Lastly, the conclusion of this study is shown.

TLR, SLR, and Delphi Method

Differences between SLR, TLR, and Delphi Method

TLR, SLR, and Delphi Method are distinct research techniques that differ in their approaches. Their differences come from their information sources, methods of data collection, and focus.

The information source for TLR and SLR comes from secondary sources, as it involves extracting data from existing research and literature (Kitchenham et al., 2016; Li & Wang, 2018; Snyder, 2019). Meanwhile, Delphi Method information source are primary sources, as it involves direct access to the expert respondents to get the input and information needed (Barrios et al., 2021).

TLR method of data collection involves summarizing and synthesizing existing research studies and other scholarly literature on a topic which may follow certain steps (Li & Wang, 2018). On the other hand, SLR instead systematically identifies, evaluates, and synthesize research activities using protocols such as the Kitchenham protocol (Kitchenham et al., 2016; Snyder, 2019). Meanwhile, Delphi Method collects data from experts and stakeholders to obtain and synthesize expert opinion and judgement (Smarandache, et al., 2020; von der Gracht, 2012).

The focus of TLR is on identifying pattern, gaps and trends in the literature and providing a comprehensive overview of the current state of knowledge on a topic (Li & Wang, 2018). SLR instead focuses on analyzing and synthesizing existing research data and results to answer a specific research question (Kitchenham et al., 2016; Snyder, 2019). Meanwhile, Delphi Method focus is on using a structured process of gathering and synthesizing experts’ opinions and judgements on a particular topic or problem (Smarandache, et al., 2020; von der Gracht, 2012).

From this, it could be inferred that TLR, SLR, and Delphi Method are research techniques that uses different approaches in order to collect and synthesize data. However, each technique contributes in an important way in providing answers on research.

Similarities between SLR, TLR, and Delphi Method

TLR, SLR, and Delphi Method are distinct research techniques however there are similarities between them. These similarities come from their usage of expert input, iterative process, and synthesis of findings.

Though the methods in which they gather inputs from experts are different with TLR and SLR gathering inputs from literature while Delphi method directly gather inputs from expert’s respondents. TLR, SLR, and Delphi Method rely on input from experts and stakeholders to inform the research process (Kitchenham et al., 2016; Li & Wang, 2018; Smarandache et al., 2020; von der Gracht, 2012).

Besides that, TLR, SLR, Delphi Method use an iterative process in order to gather and synthesize data (Barrios et al., 2021; Kitchenham, 2016; Li & Wang, 2018; von der Gracht, 2012). TLR and SLR uses an iterative process in extracting input from literature in order to get information. Meanwhile, the Delphi Method uses an iterative process in order to solicit inputs from experts through multiple rounds to get information.

All three TLR, SLR, Delphi Method aims to synthesize findings from multiple sources of information (Barrios et al., 2021; Li & Wang, 2018; Snyder, 2019). TLR and SLR synthesize input from literature in order to ascertain the current landscape of the research area and answer specific research questions. Meanwhile, Delphi Method synthesize inputs from experts in order to find consensus and stability of experts’ opinions.

Adopted Methodology

The goal of this study is 1) to examine and discover whether TLR and SLR are appropriate techniques to be used in a Delphi study and 2) construct a Delphi study process overview. Investigation into the TLR and SLR techniques is intended to provide info towards the usability and outcome of these techniques. To reach the results desired, the authors has analyzed different cases(studies) that have used TLR or SLR in a Delphi study. The studies found were from eminent journal and conferences cited by Scopus. The format of data analysis for these papers can be explained in the following steps:

(1) The data on how TLR and SLR were used in these studies were extracted

(2) The data extracted was analyzed in order to discover the results of the TLR and SLR

Following these steps and gaining the results of the analysis, an overview for a Delphi study was constructed.

Results

Use Cases

A total of seven articles from the year 2016 to 2021 were found based on our criteria of Delphi studies using TLR and SLR as a technique as shown in Table 2. Furthermore, Table 3 shows how TLR or SLR were used in the cases and what these techniques had contributed to their Delphi studies.

Table 2.

Delphi Studies.

Paper ID	Paper Name	Author	Usage	Year	Journal/Conference
L1	Integrating factor analysis and the delphi method in scenario development: A case study of dalmatia, Croatia	Kaufmann	TLR	2016	Applied geography
L2	Establishing key criteria to define and compare models of specialist palliative care: A mixed-methods study using qualitative interviews and delphi survey	Firth et al.	TLR	2019	Palliative medicine
L3	Registered nurses in expanded roles improve care in nursing homes: Swiss perspective based on the modified delphi method	Basinska et al.	SLR	2021	Journal of advanced nursing
L4	Establishment of a clinician-led guideline on the diagnosis and treatment of hirayama disease using a modified delphi technique	Lyu et al.	TLR	2020	Clinical neurophysiology
L5	Research framework of evaluation model to assess the effectiveness of coordination processes in global software development projects	Subbarao and Mahrin	SLR	2020	Journal of physics: Conference series
L6	Lessons learned from a two-round delphi-based scenario study	Schmalz, Spinler, and Ringbeck	TLR and SLR	2021	MethodsX
L7	Defining a positive work environment for hospital health care professionals: A delphi study	Maassen et al.	SLR	2021	PLoS ONE

Table 3.

TLR and SLR usage in Delphi Studies.

Paper ID	Usage	TLR/SLR Contribution
L1	The study had combined factor analysis and delphi for scenario development. The study had used an extensive TLR as the first step in their study and factor analysis procedure. The results of the TLR and factor analysis were then used in a questionnaire in the delphi phase.	1) Initial findings 49 variables that was used in factor analysis phase of the study2) Initial findings of eight groups of variables that was needed to be considered in the study.
L2	The study had combined semi-structured interview, delphi, and structured interviews in order to establish key criteria of specialist palliative care. The first step of the study was a rapid scoping review that could help to define models of specialist palliative care,	1) A topic guide was constructed covering 28 preliminary criteria.
L3	The study had used a modified delphi method, the RAND/UCLA appropriateness method as the study design in order to find relevant advanced practice nurses (APN) competencies. They had used a critical literature review (SLR) to define possible competencies and outcomes for their study.	1) From the SLR conducted relevant APN role competencies and outcome were found.
L4	The study had used a modified delphi technique in order to established a guideline on the diagnosis and treatment of hirayama disease (HD). The study had used TLR as method in order to construct a protocol for the diagnosis, treatment, and follow-up assessment of HD. They had then conducted a 10-panel expert session in order the discuss the result of the TLR	1) A protocol for the diagnosis, treatment, and follow-up assessment of HD was constructed.2) The TLR had also helped the researchers to identify the experts needed for the delphi phase.
L5	The study had used delphi to gain consensus among experts about specific strategies and related indicators that can contribute to effective coordination in global software development (GSD) projects. Before, the delphi was conducted a SLR and semi-structured interview was conducted to discover a list of coordination strategies and related indicators for GSD projects in both theoretical and exploratory perspective.	1) A list of coordination strategies and related indicators for GSD were discovered in the perspective of a theoretical study.
L6	The study had use delphi to examine future door-to-door (D2D) passenger air travel in 2035. Before the delphi phase of the study was conducted, the researchers had used TLR, SLR and expert interviews to developed key points and draft projections for the delphi questionnaire.	1) Helped the researchers to formed a basis for developing the study projections.2) Helped the researchers to gain insights into potential mobility scenario and trend.
L7	The study had used delphi in order to gain consensus on the concept of a positive work environment for hospital healthcare professionals. To generate the items for the delphi study, a SLR and semi-structured interview with experts were conducted.	1) The SLR had revealed 228 aspects that belonged to 48 work environment element that is related to work environment in a hospital healthcare setting.

Delphi Study Processes with TLR and SLR

It was also found that there are two types of processes in which TLR and SLR results are incorporated into the Delphi studies as can be seen in Figures 1, and 2. Figure 1 shows a process in which TLR and SLR are sequentially done with another technique before the Delphi phase of the study. Meanwhile, Figure 2 shows a process in which the results of the SLR and TLR were consolidated with the result of another technique before the Delphi phase.

Figure 1.

Sequential incorporation.

Figure 2.

Consolidated incorporation.

Discussion

Are TLR and SLR techniques appropriate to be use in a Delphi study?

Through the use cases which uses TLR and SLR as technique in a Delphi study that has been shown, two questions had been asked by the researcher in regards to the appropriateness of the usage of TLR and SLR in a Delphi study as shown in Table 4. The questions were answered based on the TLR and SLR contribution of the study.

Table 4.

Question Presented by Researcher of This Study.

Questions/Paper ID	1	2	3	4	5	6	7
1) Did TLR and SLR added information to the delphi study?	Yes	Yes	Yes	Yes	Yes	Yes	Yes
2) Did TLR and SLR conducted benefited the delphi study?	Yes	Yes	Yes	Yes	Yes	Yes	Yes

As can be seen in Table 4, the results show that by conducting TLR and SLR, the researchers of the use cases were able to add information and gain benefit for their Delphi study. Besides that, one similarity between all these studies is that all of them used TLR and SLR as an initial technique to discover information such as variables, indicators and etc related to their study. From this, it can be found that SLR and TLR are techniques that can provide researchers of a Delphi study with baseline or primary understanding of their study. Because of this, it can be found that TLR and SLR are indeed appropriate techniques to be used in a Delphi study.

Delphi Study Overview Construction

Through the results found, it can also be seen that there are different types of processes in which TLR and SLR can be incorporated in a study. L1, L2, and L4 had incorporated TLR and SLR by sequentially using them with another technique. They had used them sequentially by using the results of the TLR and SLR in the proceeding technique which can be either a quantitative or qualitative technique.

Another way that TLR and SLR had been incorporated in a Delphi study as shown by L3, L5, L6, and L7 is by consolidation. The researchers that had used this type of process had initially conducted a TLR or SLR as the initial technique of their Delphi study followed by using another technique to gain another perspective of their study. The results of the TLR or SLR were then consolidated with the results of their second technique.

Through the information found from these two types of processes, this study had adapted the processes and constructed a Delphi study process as can be seen in Figure 3. The Delphi process of this study will be for the topic of “Risk Management of Cloud Computing Projects in Healthcare”. The results of this process will be used to construct a risk model software. The Delphi process constructed consist of a preliminary findings stage and Delphi technique stage with the techniques being used throughout stages being TLR, SLR, constant comparison method (CCM), semi-structured interview and questionnaire/checklist.

Figure 3.

Delphi process overview.

Preliminary Findings Stage

A TLR and SLR in conjunction with CCM will be conducted in order to find relevant indicators for our study. The TLR will be used to discover and analyse technical report and standards related to the topic of this study. Furthermore, a SLR in conjunction CCM will be conducted in order to discover and analyse academic papers related to this study. The results from the TLR and SLR will then be consolidated in order to be used in Delphi technique stage.

Delphi Techniques Stage

A three-round Delphi process will be conducted. The rounds consist of a questionnaire/checklist and semi-structured interview. The questionnaire/checklist will be the method to conduct close-ended questions meanwhile semi-structured will be the method to conduct open-ended questions. The results from the three-rounds of Delphi that will be conducted, will be consolidated at the end of the rounds which will then be used to construct a risk model software.

Conclusion

To conclude, the aim of this study was 1) to examine and discover whether TLR and SLR are appropriate techniques to be used in a Delphi study and 2) to construct a Delphi study overview. The first goal was achieved through the discovery of the usage and contribution of TLR and SLR in the Delphi study by these techniques being able to add information and benefit the Delphi study even with their differences. The second aim of this study was also achieved through the discovery of the two types of processes in which TLR and SLR can be incorporated in a Delphi study which are sequential and consolidation. Through this discovery a Delphi study overview was constructed in which had adapted the abovementioned processes. We hope that the results of this study will be able to help researchers who are interested in doing a Delphi study to know on what are the benefits of TLR and SLR in these studies along with how to incorporate them.

Moving forward, we will be conducting our study using proposed Delphi process overview in our study. Through the usage of this overview, we hope to gain more insight regarding the techniques that will be used throughout our study. We hope that more researchers who are considering on doing a Delphi study will consider incorporating TLR and SLR in their study due to the benefit that they bring.

Footnotes

Acknowledgments

We are grateful to Multimedia University,Faculty of Management,family members,and other individuals for their kind support and encouragement. We are also grateful to Multimedia University for providing us with the opportunity to conduct this research and for all of the resources and support provided.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) received no financial support for the research,authorship,and/or publication of this article.

ORCID iD

Muhammad Afif Fathullah

Anusuyah Subbarao

Saravanan Muthaiyah

References

Barrios

Guilera

Nuño

Gómez-Benito

(2021). Consensus in the delphi method: What makes a decision change? Technological Forecasting and Social Change, 163, 120484. https://doi.org/10.1016/j.techfore.2020.120484

Basinska

Wellens

N. I. H.

Simon

Zeller

Kressig

R. W.

Zúñiga

(2021). Registered nurses in expanded roles improve care in nursing homes: Swiss perspective based on the modified delphi method. Journal of Advanced Nursing, 77(2), 742-754. https://doi.org/10.1111/jan.14644

Firth

A. M.

O’Brien

S. M.

Guo

Seymour

Richardson

Bridges

Hocaoglu

M. B.

Grande

Dzingina

Higginson

I. J.

Murtagh

F. E. M.

(2019). Establishing key criteria to define and compare models of specialist palliative care: A mixed-methods study using qualitative interviews and delphi survey. Palliative Medicine, 33(8), 1114-1124. https://doi.org/10.1177/0269216319858237

Kitchenham

B. A.

Budgen

Brereton

(2016). Evidence based software engineering and systematic reviews. CRC Press.

Wang

(2018). Traditional literature review and research synthesis. In Phakiti

De Costa

Plonsky

Starfield

(Eds.), The palgrave handbook of applied linguistics research methodology. Palgrave Macmillan. https://doi.org/10.1057/978-1-137-59900-1_6

Lyu

Zheng

Wang

Nie

Xia

Zhu

Jin

Sun

Zhu

Kuwabara

Cortese

Maqbool Hassan

Takai

Paredes

Webere

Turk

Kimura

Jiang

(2020). Establishment of a clinician-led guideline on the diagnosis and treatment of Hirayama disease using a modified Delphi technique. Clinical Neurophysiology, 131(6), 1311-1319. https://doi.org/10.1016/j.clinph.2020.02.022

Maassen

S. M.

van Oostveen

Vermeulen

Weggelaar

A. M.

(2021). Defining a positive work environment for hospital healthcare professionals: A delphi study. PLoS One, 16(2 February), 1-14. https://doi.org/10.1371/journal.pone.0247530

Nasa

Jain

Juneja

(2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116-129. https://doi.org/10.5662/wjm.v11.i4.116

Radeljak Kaufmann

(2016). Integrating factor analysis and the delphi method in scenario development: A case study of dalmatia, Croatia. Applied Geography, 71, 56–68. https://doi.org/10.1016/j.apgeog.2016.04.007

10.

Schmalz

Ringbeck

Spinler

(2020). Door-to-door travel in 2035 – a Delphi study. Technological Forecasting and Social Change, 157, 120096. https://doi.org/10.1016/j.techfore.2020.120096

11.

Schmalz

Ulrike

Spinler

Stefan

Ringbeck

Jürgen

(2021). Lessons Learned from a Two-Round Delphi-based Scenario Study. MethodsX, 8, 101179. https://doi.org/10.1016/j.mex.2020.101179

12.

Smarandache

Ricardo

J. E.

Caballero

E. G.

Vázquez

M. Y. L.

Hernández

N. B.

(2020). Delphi method for evaluating scientific research proposals in a neutrosophic environment. Neutrosophic Sets and Systems, 34, 204–213. https://doi.org/10.5281/zenodo.3820430

13.

Snyder

(2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 104, 333–339. https://doi.org/10.1016/j.jbusres.2019.07.039

14.

Subbarao

Mahrin

M. N.

(2020). Research framework of evaluation model to assess the effectiveness of coordination processes in global software development Projects. Journal of Physics: Conference Series, 1529(5), 052064. https://doi.org/10.1088/1742-6596/1529/5/052064

15.

von der Gracht

H. A.

(2012). Consensus measurement in Delphi studies. Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525-1536. https://doi.org/10.1016/j.techfore.2012.04.013

Methodological Investigation: Traditional and Systematic Reviews as Preliminary Findings for Delphi Technique