Modified Delphi methods are increasingly used to develop healthcare pathways with input from people with lived experience (PWLE) and clinicians/others. However, guidance on consensus analysis in this context remains limited. We examined consensus outcomes across different scoring methods and criteria when participants were treated as a single combined group (Objective 1) versus two distinct groups (Objective 2).
Methods
We conducted a secondary analysis of Round 1 data from a project involving PWLE (N = 8) and clinicians/others (N = 51). To assess agreement on 68 Delphi statements, we applied three methods for scoring percentage agreement that differed in how the middle response on a three-point Likert scale (“approve”, “not sure either way”, “do not approve”) was treated. Method 1 excluded the middle response, methods 2 and 3 grouped the middle response with “do not approve”, and “approve”, respectively. We compared consensus rates (% of items reaching consensus) using percentage agreement cutoffs of ≥70%, ≥80%, and ≥90% of participants.
Results
Consensus results varied by participants grouping, treatment of middle response categories, and cutoff criteria. Results from the combined group of PWLE and clinicians/others provided a simplified overview consensus outcome. Treating the participants into as separate groups provide nuanced results.
Conclusion
The analysis of data can change the results from which to draw conclusions and inform practice. Investigators should consider the alignment of each approach with the goals of their Delphi study.
BarringtonH.YoungB.WilliamsonP. R. (2021). Patient participation in Delphi surveys to develop core outcome sets: Systematic review. BMJ Open, 11(9), e051066. https://doi.org/10.1136/bmjopen-2021-051066
2.
BeiderbeckD.FrevelN.von der GrachtH. A.SchmidtS. L.SchweitzerV. M. (2021). Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. MethodsX, 8(101401), 1–20.https://doi.org/10.1016/j.mex.2021.101401
3.
BestR. W.CampbellJ. T.EdwardsM. C.ClineL. L. (2025). An Overview of the Delphi Method’s Origin, Modifications, and Use to Augment Instrument Development and Data Collection: A Research Note. Journal of International Agricultural and Extension Education, 32(1). https://doi.org/10.4148/2831-5960.1497
4.
BoulkedidR.AbdoulH.LoustauM.SibonyO.AlbertiC. (2011). Using and reporting the delphi method for selecting healthcare quality indicators: A systematic review. PLoS ONE, 6(6), e20476–e20476. https://doi.org/10.1371/journal.pone.0020476
5.
BrookesS. T.MacefieldR. C.WilliamsonP. R.McNairA. G.PotterS.BlencoweN. S.StrongS.BlazebyJ. M. (2016). Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials, 17(1), 409. https://doi.org/10.1186/s13063-016-1479-x
6.
CampbellH.HotchkissR.BradshawN.PorteousM. (1998). Integrated care pathways. British Medical Journal, 316(7125), 133–137. https://doi.org/10.1136/bmj.316.7125.133
de LoëR. C.MelnychukN.MurrayD.PlummerR. (2016). Advancing the state of policy Delphi practice: A systematic review evaluating methodological evolution, innovation, and opportunities. Technological Forecasting and Social Change, 104, 78–88. https://doi.org/10.1016/j.techfore.2015.12.009
9.
DiamondI. R.GrantR. C.FeldmanB. M.MooreA. M.WalesP. W.PencharzP. B.LingS. C. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409.https://doi.org/10.1016/j.jclinepi.2013.12.002
10.
EubankB. H. F.LackeyS. W.SlompM.WerleJ. R.KuntzeC.ShepsD. M. (2021). Consensus for a primary care clinical decision-making tool for assessing, diagnosing, and managing shoulder pain in Alberta, Canada. BMC Family Practice, 22(1), 1–20.https://doi.org/10.1186/s12875-021-01544-3
11.
EverinkI. H. J.van HaastregtJ. C. M.KempenG. I. J. M.ScholsJ. M. G. A. (2020). Building consensus on an integrated care pathway in geriatric rehabilitation: A modified Delphi study among professional experts. Journal of Applied Gerontology, 39(4), 423–434. https://doi.org/10.1177/0733464818774629
12.
FernandezL.ThompsonS.BerendonkC.Schick-MakaroffK. (2022). Mental health care for adults treated with dialysis in Canada: A scoping review. Canadian Journal of Kidney Health and Disease, 9, 1–16. https://doi.org/10.1177/20543581221086328
13.
GattrellW. T.HunginA. P.PriceA.WinchesterC. C.ToveyD.HughesE. L.van ZuurenE. J.GoldmanK.LogulloP.MatheisR.HarrisonN. (2022). ACCORD Guideline for reporting consensus-based methods in biomedical research and clinical practice: A study protocol. Research Integrity and Peer Review, 7(1), 1–10.https://doi.org/10.1186/s41073-022-00122-0
14.
HassonF.KeeneyS. (2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change, 78(9), 1695–1704. https://doi.org/10.1016/j.techfore.2011.04.005
HoI. S. S.Azcoaga-LorenzoA.AkbariA.DaviesJ.KhuntiK.KadamU. T.LyonsR. A.McCowanC.MercerS. W.NirantharakumarK.StaniszewskaS.GuthrieB. (2022). Measuring multimorbidity in research: Delphi consensus study. BMJ Medicine, 1(1), e000247. https://doi.org/10.1136/bmjmed-2022-000247
17.
Hock OngM. E.ShinS. D.SungS. S.TanakaH.Huei-MingM.SongK. J.NishiuchiT.LeongB. S.-H.KarimS. A.LinC.-H. (2013). Recommendations on ambulance cardiopulmonary resuscitation in basic life support systems. Prehospital Emergency Care, 17(4), 491–500. https://doi.org/10.3109/10903127.2013.818176
18.
HsuC.-C.SandfordB. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation, 12(1), 1–8. https://doi.org/10.7275/pdz9-th90
19.
KeeneyS.HassonF.McKennaH. P. (2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38(2), 195–200. https://doi.org/10.1016/S0020-7489(00)00044-4
KhodyakovD.GrantS.DengerB.KinnettK.MartinA.PeayH.CoulterI. (2020). Practical considerations in using online modified-Delphi approaches to engage patients and other stakeholders in clinical practice guideline development. The Patient - Patient-Centered Outcomes Research, 13(1), 11–21. https://doi.org/10.1007/s40271-019-00389-4
LangeT.KopkowC.LütznerJ.GüntherK.-P.GraviusS.ScharfH.-P.StöveJ.WagnerR.SchmittJ. (2020). Comparison of different rating scales for the use in Delphi studies: Different scales lead to different consensus and show different test-retest reliability. BMC Medical Research Methodology, 20(1), 28. https://doi.org/10.1186/s12874-020-0912-8
24.
LiJ.AroojisA.MulpuriK.SheaK. G.SchaefferE. K. (2021). Development of a DDH care pathway for India: A study methodology to guide similar efforts in other countries and for other conditions. Indian Journal of Orthopaedics, 55(6), 1549–1558. https://doi.org/10.1007/s43465-021-00534-y
25.
NasaP.JainR.JunejaD. (2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116–129. https://doi.org/10.5662/wjm.v11.i4.116
PanellaM.MarchisioS.Di StanislaoF. (2003). Reducing clinical variations with clinical pathways: Do pathways work?International Journal for Quality in Health Care, 15(6), 509–521. https://doi.org/10.1093/intqhc/mzg057
31.
Pedrosa CarrascoA. J.BerlinP.BetkerL.Riera-KnorrenschildJ.von BlanckenburgP.SeifartC. (2022). Developing a care pathway for hospital-based advance care planning for cancer patients: A modified Delphi study. European Journal for Cancer Care, e13756. https://doi.org/10.1111/ecc.13756
Schick-MakaroffK.BerendonkC.OverwaterJ.StreithL.LeeL.EscotoM.CukorD.KlarenbachS.SawatzkyR. (2022). How are Albertans “adjusting to and coping with” dialysis? A cross-sectional survey. Canadian Journal of Kidney Health and Disease, 9, 20543581221118436. https://doi.org/10.1177/20543581221118436
34.
Schick-MakaroffK.BerendonkC.SalumM.YoeunP.WichartJ.ArmstrongM.ThompsonS.ElliottM.LeeL.SmithT.ReintjesF.FillierD.KlarenbachS.SawatzkyR. (2025). Developing and tailoring a person-centred pathway for mental health care for people receiving dialysis. Nephron, 149(7), 392–410.https://doi.org/10.1159/000544058
ShawJ.PriceM.ClaytonJ.GrimisonP.ShawT.RankinN.ButowP.ShawJ. M.PriceM. A.ClaytonJ. M.ButowP. N. (2016). Developing a clinical pathway for the identification and management of anxiety and depression in adult cancer patients: An online Delphi consensus process. Supportive Care in Cancer, 24(1), 33–41. https://doi.org/10.1007/s00520-015-2742-5
37.
SossaJ. W. Z.HalalW.Raul HernandezZ. (2019). Delphi method: Analysis of rounds, stakeholder and statistical indicators. Foresight: The Journal of Futures Studies, Strategic Thinking and Policy, 21(5), 525–544. https://doi.org/10.1108/FS-11-2018-0095
38.
SpitzerR. L.KroenkeK.WilliamsJ. B.LöweB. (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. https://doi.org/10.1001/archinte.166.10.1092
39.
StewartD.Gibson-SmithK.MacLureK.MairA.AlonsoA.McIntoshJ.CodinaC.CittadiniA.Fernandez-LlimosF.FlemingG.GennimataD.GillespieU.KempenT.HarrisonC.Junius-WalkerU.WieseB.KardasP.LewekP.KinnearM.…ScullinC. (2017). A modified delphi study to determine the level of consensus across the European Union on the structures, processes and desired outcomes of the management of polypharmacy in older people. PLoS ONE, 12(11), e0188348. https://doi.org/10.1371/journal.pone.0188348
40.
ThompsonL.HillM.LeckyF.ShawG. (2021). Defining major trauma: A Delphi study. Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine, 29(1), 63. https://doi.org/10.1186/s13049-021-00870-w
41.
von der GrachtH. A. (2012). Consensus measurement in Delphi studies. Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536. https://doi.org/10.1016/j.techfore.2012.04.013
42.
WatanabeS. M.NekolaichukC.BeaumontC.JohnsonL.MyersJ.StrasserF. (2011). A multicenter study comparing two numerical versions of the Edmonton symptom assessment system in palliative care patients. Journal of Pain and Symptom Management, 41(2), 456–468. https://doi.org/10.1016/j.jpainsymman.2010.04.020
43.
WoodcockT.AdelekeY.GoeschelC.PronovostP.Dixon-WoodsM. (2020). A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects. BMC Medical Research Methodology, 20(1), 8. https://doi.org/10.1186/s12874-019-0886-6
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
