Sage Journals: Discover world-class research

Abstract

Objectives

The aim of the study is to describe the user experiences of a nationwide digital decision support system (DDSS).

Summary of background data

DDSSs have the potential to improve the quality and safety of healthcare services by supporting clinical decision-making with evidence-based recommendations. Due to a lack of knowledge, it is difficult to assess whether DDSSs are fulfilling their purpose. In Estonia, a nationwide DDSS for general practitioners (GPs) was implemented in 2020. To understand the impact of DDSS on the quality of care in the Estonian context and meet the demands of healthcare, it is necessary to gather information about the experiences of the users. This is the first study that examines the experiences of GPs on the use of DDSS nationwide.

Methods

A qualitative descriptive study was conducted based on snowball sampling. Semi-structured interviews were performed in February–March 2022 with nine GPs. Data were analyzed by thematic analysis. A total of six themes and 16 subthemes emerged from the data.

Results

A total of six themes and 16 subthemes emerged from the data. The following themes were identified: user-friendliness, DDSS use in clinical practice, benefits of the DDSS, and the impact of the DDSS on GPs’ work, barriers to using the DDSS, and suggestions for improving the user experience. The results of the study are important, as they address and contribute to the relevant aspects of digital health in primary care.

Conclusion

GPs shared their individual user experiences, including user-perceived barriers and enabling factors that influence the implementation and use of a decision support system in primary care settings. It is revealed that GPs have different benefits and barriers depending on the topic discussed. Future research should evaluate the functioning of the DDSS and the quality of the decisions it provides by observing and evaluating patient records. Systematic user experiences need to be collected and examined to ensure the usability and sustainability of the DDSS.

Keywords

Digital decision support system evidence-based personalized medicine user experience primary care qualitative study

Introduction

Digital decision support systems (DDSSs) are software solutions developed to facilitate clinical decision-making, where patients’ health data are matched with computerized clinical knowledge or a machine learning (ML) algorithm, and then personalized recommendations are presented to inform decision-making.^1,2 Different types of DDSSs are widely used at different levels within the healthcare system to support specific processes of care and health issues, both in medical and nursing care. A DDSS is a crucial tool that primarily deals with health and medical data, and recently, genome data as well.^1–7

A DDSS is used to support the clinician's decision-making process in several ways. It can improve the treatment process by influencing the quality of treatment, reducing treatment errors, increasing healthcare professionals’ adherence to treatment instructions, and improving patient outcomes/health outcomes involving patients’ data assessment.^6–11

There are knowledge-based and non-knowledge-based DDSSs.⁹ Knowledge-based systems are based on rules (IF-THEN statements), and the system retrieves data to evaluate each rule and produce an action or output (Figure 1). Non-knowledge-based DDSSs produce recommendations based on different artificial intelligence methods, such as ML, or statistical design, rather than evidence-based (EB) knowledge.

Figure 1.

Key interactions in a knowledge-based digital decision support system (DDSS).⁹

Decision-making support can allow healthcare professionals to work easier and faster. However, one of the important aspects of using DDSS is a positive user experience. Unfortunately, based on the literature there is no international consensus regarding approved methods of measuring the user experiences of clinical decision support tools.^11–17 Therefore, it is difficult to confirm whether the decision support systems are effective and useful (i.e. whether they fulfill their purpose, meet user expectations, and what is the quality of their outputs).

Implementation of DDSSs in the Estonian Nationwide Health Information System

Primary care is a gatekeeper of patient journeys in the Estonian healthcare system, which is based on the solidarity-based principle, meaning that all insured people in Estonia are entitled to the same quality healthcare. General practitioners (GPs) have free access to the Evidence-Based Medicine Electronic Decision Support System (EBMeDS) that is implemented nationwide. It is used for clinical decision-making through patient-specific recommendations and entries both in real-time and for special groups of patients.¹⁸

EBMeDS is a unique, simple, and structured knowledge-based software solution, which was developed by the Finnish publisher Duodecim Medical Publications and has been implemented in Estonia since 2020. EBMeDS supports qualitatively new primary care services by speeding up the decision-making process. It is registered as a medical device in Estonia.^19,20

The uniqueness of the Estonian setup is that the system is a nationwide solution, and all patients’ health data across the country are considered. Likewise, EBMeDS uses a patient-oriented Estonian Nationwide Health Information System (EHIS) as an Electronic Health Records (EHR) service, in which healthcare providers at different healthcare levels have exchanged patients’ health data since 2008. EHIS includes a medical overview of visits, anamnesis, diagnoses, received treatment, and recommendations, which is visible to all healthcare professionals taking care of the patient.¹⁸ The data in EHIS is partly in the structured form using international classifiers and terminologies (International Classification of Diseases 10th Revision (ICD-10), LOINC, ATC, etc.) and partly in the free text. For data exchange, international standards, HL7 Clinical Document Architecture, and Fast Healthcare Interoperatability Resources are used.²¹

All healthcare service providers in Estonia are obliged to send certain data and documents defined by law to the EHIS. The data are also visible to the patient through the patient portal. EBMeDS is connected to all electronic medical records (EMRs) used by GPs and to the Prescription Center, which is a central service for e-prescriptions. There are five different EMRs for GPs in use in Estonia. EBMeDS automatically queries and analyses the data from EHIS, the Prescription Center, and the EMR of the GP and presents reminders, therapeutic suggestions, and links to patient-specific guidelines through the EMR user interface. The aim is to offer users the most recent medical information relevant to the present treatment context, supporting their clinical performance. Both the medical data and data exchange standards are matched to ensure that decision algorithms receive information in a structured and standardized format.^20–22

EBMeDS analyzes patient data, compares the patient's status to criteria based on clinical guidelines, or looks for any inconsistencies in the data. Based on the analysis, reminders or alerts are created and sent to the GP's desktop. EBMeDS supports decision-making in 27 specialties and can prefill interactive electronic forms with patient data, such as calculators, recommendations, or algorithms.^20,21

According to the register of the Health Board Republic of Estonia, a total of 1195 GPs are registered in Estonia.²³ Most Estonian GPs (80%) use the DDSS in daily practice, and for 95% of GPs, the DDSS is available on their desktop.²⁴

User experiences of a DDSS

Previous studies showed positive user experiences of GPs due to access to drug interactions and calculations, EB up-to-date guidelines, and alertness provided by the EBMeDS, which drew attention to issues that were not always being paid attention to.¹⁶ GPs experienced several obstacles and negative experiences related to technical problems, irrelevant reminders, not up-to-date information, and lack of time for using EBMeDS which encouraged ignoring reminders.^16,19

Evidence shows that alerts, reminders, or feedback provided by decision support systems to healthcare professionals can influence patient care (e.g. diabetes care).¹² Koskela et al.¹⁶ investigated individually perceived barriers and enabling factors affecting the implementation and use of a DDSS by healthcare professionals. Their study revealed that the reminders transmitted by the system are effective in shaping the treatment management behaviors of doctors.¹⁶

It is assumed that the entry of the GP in the health record affects the rules/suggestions offered by the decision support system, which in turn affects the quality of treatment. Marcolino et al.¹³ evaluated the perceived feasibility, usability, and usefulness of a decision support system and the satisfaction of healthcare workers six months after the implementation of the system. In addition, two focus group interviews were conducted with users: GPs and nurses.^13,15 In terms of feasibility, both GPs and nurses agreed that decision support can be used in primary care. Regarding ease of use, they stated that the decisions provided by DDSS were simple to use and particularly useful in carrying out preventive activities and facilitating the treatment process.¹³

There are several advantages of the DDSS, including reducing medication errors, improving patients’ safety, clinical management and documentation, and supporting diagnostics. The following risks are associated with the DDSS: disrupted workflow, unnecessary alerts and notifications, interoperability issues, data content and quality, lack of standardized metrics, affordability, and other concerns.⁹

Despite the benefits, little is known about the experiences of users. Thus, the results of this research will be helpful for improving the implementation of the DDSS by GPs and the quality of treatment at the primary care level when using technology to support clinical decision-making. To understand the quality of decision support in the Estonian context, it is necessary to evaluate both the functioning of the system as a whole and the experiences of the users. This was the first study to evaluate the user experience of a DDSS in response to the expectations of healthcare providers. This research is important, as it addresses and contributes to the relevant aspects of digital health in primary care. Accordingly, the aim of the study was to describe the user experiences of the DDSS and gain a sense of the underlying patterns and reasons GPs have for interacting with it.

Methods

A qualitative descriptive study was conducted, with snowball sampling used to recruit the interviewees. Initially, the plan was to conduct 10 interviews or to collect data until the database became saturated, that is, when no new information was added during the interviews and the answers began to repeat.²⁵ For this purpose, the Estonian Society of General Practitioners (EPS) was approached with a request to distribute an invitation to participate in the study, because the majority of GPs are members of the EPS. The inclusion criteria were as follows:

Willing to participate in the study and share user experiences.

Registered as a GP in the registry of the Estonian Health Board.

A member of the EPS.

Working as a GP with one's own or another GP's patientś list.

Using the EBMeDS in her/his daily clinical work.

Hence the exclusion criteria were:

Being unable to give informed consent.

Not registered in the registry of the Estonian Health Board.

Does not work permanently with a patients’ list.

Does not have access to the decision support system and, therefore cannot work with it daily.

The GPs who agreed were then asked to recommend other contacts who fit the research criteria and who might be willing to participate, and in turn, who could also recommend other potential participants. The reason for utilizing a qualitative study design and snowball sampling was that it was difficult to objectively calculate the actual accuracy and impact of the DDSS. Due to a lack of research data, it is unclear whether the DDSS is fulfilling its purpose and the quality of the decisions, alerts, and reminders it provides. In order to evaluate the recommendations offered by the DDSS and their compliance with the physician, the patient's health and related sensitive data from the EMR and related documentation should be used. In other words, it can be challenging from a data protection perspective.²⁶

Data were collected in February–March 2022. A total of nine GPs were interviewed. GPs who agreed to participate in the interview contacted the researcher using the contact details listed in the invitation, and then a suitable time and place for the interview (on-site or online/MS Teams, Zoom, Skype) were agreed upon.

Informed consent was obtained from all participants.²⁷ Since all the GPs preferred video interviews (using MS Teams), digital signing of the informed consent form was used, as electronic signing allows document files to be signed without paper and saves time.²⁸ Semi-structured interview questions (8) adapted from Koskela et al. were used for the interviews (after translation into the Estonian language) to clarify user-perceived barriers and enabling factors that influence the implementation and use of a decision support system by healthcare professionals.¹⁶ The interview questions were as follows:

Please share the user experience of the DDSS?

In which clinical situation did you use decision support? How much time did it take? What was the consequence of this?

What do you think are the best elements of this decision support?

What impact has decision support had on your work or collaboration between other professionals and patients? Has it affected the way you work?

Which decision support elements have you paid attention to? Which ones have you not?

Have you encountered obstacles when using decision support?

What ideas/suggestions do you have for improving decision support?

Is there anything else about the decision support system that we haven’t mentioned yet?

During the interviews, the respondents were directed to delve more into the topics with guiding questions: “Please give an example, how was it exactly, can you add something else, what it meant to you, how it affected you.” At the end of each interview, the interviewee was given the opportunity to add something if desired.

Data analysis

The duration of the interviews was 30–45 min, which were conducted as video meetings. All the interviews were recorded in MS Teams and transcribed into text by the lead author after the interview was finished. All interviews were then listened to and read over, avoiding the loss of text, and then the recordings were deleted from the recorder. No personally identifiable information was transcribed during the interviews.

The data were analyzed based on a six-step thematic analysis framework, which is a qualitative data analysis method for identifying themes within qualitative data.^29–31 In the first step, the researcher became familiar with the data, and important notes were recorded. Step two included the generation of the initial codes. Open coding was used, meaning codes were developed during the coding process. In the third step, a search for themes was conducted. Descriptive initial themes describing patterns in the data relevant to the research questions were identified. In step four, the initial themes were reviewed and modified. In step five, the final refinement of the themes was conducted. The process ended with a write-up in step six.^29–31 The result was based on 26 codes, which were categorized into six themes and 16 subthemes (see Table 1). The presentation of the findings is based on the themes, subthemes, and quotations.

Table 1.

Main themes and subthemes identified.

Themes	Subthemes
User friendliness	Positive and negative user experiences
DDSS use in clinical practice	Patient with multimorbidity
DDSS use in clinical practice	Adult health examination
Benefits of the DDSS	Modern solution
	Supports clinical decision-making
	Digital solution integrated with GPs dashboard
	Convenient to use
Impact of the DDSS on GPs’ work	Time saving
	Improved communication
	Improved documentation
Barriers to using the DDSS	Mismatched input data
Suggestions for improving the user experience	Warning when entering wrong information
	Warning if data differs from reference values
	Compliance of data with system recommendations
	Data exchange when changing the patient
	Timing of alerts/warnings

DDSS: digital decision support system; GP: general practitioner.

Results

The analysis revealed six main themes: user-friendliness, DDSS use in clinical practice, benefits of the DDSS, the impact of the DDSS on GPs’ work, barriers to using the DDSS, and user experience improvement suggestions (Table 1).

The first identified theme was “user-friendliness,” based on GPs’ shared user experience of the DDSS. There were both positive and negative experiences shared by GPs, who reported that the DDSS was “convenient to use” (GP1, GP2, GP4, GP5, GP6, GP7, GP9) and “simple.” (GP2, GP4, GP5, GP6, GP7, GP9). As with any innovation, the users expressed some initial bias: “At first it seemed complicated, and I was confused. I needed time to learn how to use it.” (GP3). It was also reported that the DDSS operates “…slowly, wastes the time of the visit…” (GP8). The second identified theme, “DDSS use in clinical practice,” referred to the use of the system in daily practice with the patient. The majority of patients GPs are taking care of in their daily practice have several chronic diseases (multimorbidity). GPs shared that DDSS was used for “…working with chronically ill patients…,” (GP8, GP1, GP3, GP5), as well as for adult patients who required a license for “…driving a motor vehicle” (GP4). One of the interviewees used the DDSS with a “… regular patient arrived for a visit…” (GP1–GP9) and found the system to be “…time-saving” (GP1, GP2, GP4, GP7, GP9).

Third, the “benefits of the DDSS” were supported by subthemes like “modern solution” (GP1, GP2, GP4, GP5, GP6), which “supports clinical decision-making” (GP7, GP9) and is “convenient to use” (GP1, GP2, GP3), because it is a “digital solution” that is “integrated with GPs’ desktop” (GP1–GP9). Based on the interviewees, the DDSS “supports GPs’ work” (GP1, GP4, GP5, GP6) by giving “suggestions based on the current treatment guidelines, no need for search anything else” (GP2, GP3), as well as providing “instructions, etc., that I would not have thought of immediately” (GP2, GP5), and recommendations for “consider making diagnostics and lab tests” (GP6, GP7, GP8). The participants stated that the current DDSS is “modern and convenient because of [its] integration with the GPs’ dashboard” (GP3, GP4).

The fourth identified theme was “impact of the DDSS on GPs’ work,” which was supported by subthemes like “time-saving” (GP1, GP2, GP3, GP5) and “improved both communication and documentation” (GP4, GP6, GP9). Using the DDSS “enables GPs to spend more time on the patient” (GP1, GP2, GP4). Moreover, the GPs stated that “documentation improved” as well as “communication among other specialists” (GP2, GP7, GP9).

The theme “barriers to using DDSS” was mainly related to “mismatched input data.” GPs reported that “When opening new patient´s health record, recommendations related to the previous patient displayed” (GP2, GP8), which in turn raises doubts about the trustworthiness and quality of the data and the system. Also, GPs pointed out that “when I search for a recommendation, this data field is empty, but when I click on it, the treatment recommendations appear” (GP8, GP9).

GPs also made suggestions related to “user experience improvement.” Several subthemes were identified, including “warning when entering wrong information” (GP2, GP3, GP8), “warning if data differ from reference values” (GP1, GP2, GP5), “compliance of data with system recommendations” (GP3, GP4, GP5, GP9), “data exchange when changing the patient” (GP8, GP9) and “timing of alerts/warnings” (GP1, GP8, GP9). When inserting incorrect clinical data, an exception should be generated and displayed, which does not allow the system to proceed further: “If there is any error in the values of the blood tests or a repetition, ex-high cholesterol values are over the reference value, then appropriate warnings or recommendations should be given/displayed” (GP8) and “While viewing the health record there is no entry in the health record about whether the doctor followed recommendations or not” (GP2, GP5, GP6). This is because it is not possible to assess whether the decision/recommendation of the decision support system was taken into account by other GPs: “The DDSS shows the recommendations given to the previous patient. Therefore, the reliability/relevance of the offered recommendation is questionable” (GP8). In this case, the warning is incomprehensible because the recommendation/warning has already been made.

Discussion

The results of this study are based on user experiences of the nationwide DDSSs, the first time that GPs’ experiences with the DDSS have been examined in Estonia. To the best of our knowledge, Estonia is the first country in which clinical DDSS is used nationwide. The current study is important, as it addresses and contributes to the relevant aspects of digital health in the Estonian primary care settings. Six main themes were revealed in the study.

The theme “barriers to using DDSS” was mainly related to “mismatched input data.” There is evidence that non-relevant recommendations, alert fatigue, a lack of user-friendliness, slow and poor integration with the EHR, and information overload affect the use of the DDSS.² Barriers revealed from this study included technical and content-related obstacles, while non-relevant recommendations and information explosion can be considered an experts’ habit phenomenon. GPs have their own professional behavior based on individual clinical intuition, expertise, and beliefs that are not always in alignment with the organizational changes, which in turn may lead to difficulties in accepting and effectively making EB clinical decisions. One possible explanation for not adopting and accepting the DDSS could be that the system does not meet the needs or support the current way of clinical working, meaning it simply is not serving its purpose.

In recent studies, several barriers have been reported related to DDSS use, such as poor digital competence, limited access to EHRs, lack of timely technical support, overloaded work environments, and poor interaction with other software solutions.⁶ In a recent study conducted by Horwood et al. (2023),¹¹ participants suggested that alerts could be displayed when incorrect data are inserted by the GP into the DDSS or if the values do not match the reference values. In this study, the GPs reported that the DDSS displays recommendations related to the previous patient, not the next patient who is visiting the GP. This could be explained by the fact that the software system is running slowly, which does not allow the comparison and alignment of data. Since the DDSS does not analyze or match free, narrative text, it is important for GPs and nurses to insert data in a structured way into the appropriate data fields, to use the correct ICD-10 codes, and to insert the correct prescription type and treatment plan when writing a prescription.

DDSSs are used worldwide in various clinical situations, and they are considered a way to improve healthcare delivery.^1–5 However, the use of a DDSS is limited to the EMRs of a healthcare provider or EHRs of a specific region. The results of our study also confirmed that the DDSS in Estonia is used to deal with various health issues due to its benefits. GPs are the gatekeepers of the healthcare system, and patients with various problems need to be effectively assessed during the limited visit time and treated appropriately. It is crucial to provide appropriate conditions at the primary level to assist and support GPs in achieving their clinical goals to improve patients’ health outcomes.

Based on the study conducted by the Estonian Health Insurance Fund,²⁴ only 80% of GPs use the EBMeDS, which is surprising since the DDSS has been enabled for several years. Similar results were reported in the study of Kortteisto et al.³² On the one hand, this may be due to the fact that GPs in Estonia have no unified EMR, and technical issues may affect the acceptance and usage of the DDSS. Fragmented digital systems may be a problem and may affect patients’ safety.¹ However, on the other hand, it may be due to GPs’ lack of knowledge or previous experience regarding using the DDSS, or it may be assumed that the DDSS itself lacks a user-friendly design. Users’ digital competency, previous experience with similar digital solutions, and appropriate training all facilitate proper usage and interaction with the DDSS.^33–35 There is a need for clear and concise training that supports GPs in the use of the system. The motivation of GPs to use the system could also be increased by confirming the evidence that informs the system, which does not replace the clinical intuition of several specialists. Since the priority of GPs’ clinical work is patients’ well-being, it would be unreasonable to waste this valuable professional resource on poor development. However, as long as the use of the system is not fully accepted by GPs and is voluntary, it cannot be ensured that the target group will use it appropriately despite its benefits.

Moreover, GPs reported the “impact of the DDSS” on their work, saving time and enhancing communication at all levels as well as improving documentation. These results offer valuable insights regarding quality-of-care coordination, which can improve long-term patient outcomes. Casey et al.⁶ reported similar findings, as the majority of physicians who participated in their study agreed that the DDSS improves patient outcomes and saves time. In a recent study by Meunier et al.,² physicians stated that using DDSS increases their self-confidence and improves care, ensuring patients receive appropriate care, including patient education.

The availability of medical care may be affected by a lack of healthcare personnel and insufficient time for patient visits, making it difficult to identify needs, make decisions, and provide documentation at the same time. Patients seeking medical help at primary healthcare centers may have several conditions simultaneously and may use different healthcare services in various healthcare institutions. It may also be the case that patient data are not available to all parties, although all relevant health data should be transferred to the central system in the Estonian case. Users prefer DDSSs that are simply designed, save time, and are easy to learn and install if needed.⁶ It is an important benefit for GPs that EBMeDS is integrated with their desktop. This saves time when dealing with data and, at the same time, harmonizes the available data and offers relevant recommendations. Despite the assumptions, it is important for clinicians that the DDSS recommendations encourage them rather than oblige them to interact. One recent study highlighted the importance of accessing the DDSS from different locations of the EHR.⁶

It is vital that the software solution is not distracting and provides evident value. When developing technical solutions, specialists working in this field should be involved in the development process, and their opinions should be considered.¹ Ineffective care coordination and the underlying suboptimal teamwork processes are healthcare organization issues. To provide safe and quality care, accurate and effective teamwork-related interventions, as well as collaboration at different levels, are required.³³^,^35,36

The current study has some limitations. Firstly, since it is a first-time study in Estonia, it is not possible to compare the results gained with those of previously published ones that emerged from the same conditions. Secondly, the results cannot be generalized to all clinical software and/or DDSSs and GPs, as this study mapped individual user experiences to a specific EBMeDS. However, the involvement of end users in the development process and systematic evaluation of usability are important inputs for the implementation and sustainable usage of the software. Thirdly, the study sample is rather poor, but the design of the study was found in prior methodological studies. The poor sample may be caused by the fact that only those GPs who are most interested in e-health and electronic data systems agreed to participate in the study. The authors would consider rearranging and reanalyzing interviews to confirm saturation in future research. Also, the focus group interview could be used as an alternative method in this study or in future studies so participants could share their thoughts and ideas regarding DDSSs.

Conclusion

GPs shared their individual user experiences including user-perceived barriers and enabling factors that influence the implementation and use of a decision support system in primary care settings. It is revealed that GPs have different benefits and barriers depending on the topic discussed. Based on the study results, the DDSS has the potential to serve this purpose, despite several developmental issues.

The study showed that certain improvements are required, and user experiences and opinions should be considered in making those improvements. In particular, future studies are needed to examine systematic user experiences and evaluate the functioning of the system and the quality of the decisions provided by the DDSS. This should involve observing and evaluating EHR as well as focusing on the pros and cons of using the DDSS. Systematic user experiences need to be collected and examined to ensure the usability and sustainability of the DDSS.

Supplemental Material

sj-pdf-1-dhj-10.1177_20552076241271816 - Supplemental material for General practitioners’ user experience of the nationwide digital decision support system in primary care

Supplemental material, sj-pdf-1-dhj-10.1177_20552076241271816 for General practitioners’ user experience of the nationwide digital decision support system in primary care by Jekaterina Šteinmiller and Peeter Ross in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors would like to thank all of the survey participants. JS would like to thank Daniel Lupshitz for his valuable expertise in finalizing this research.

Contributorship

JS researched the literature and conceived the study. JS and PR were involved in gaining ethical approval and agreement from the Association of Estonian General Practitioners. JS collected data and interviewed GPs. JS wrote the first draft of the manuscript. Both authors,JS and PR,critically reviewed and edited the manuscript and approved the final version of the manuscript to be published. Both authors participated sufficiently in the work.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Ethical Approval

The ethics committee of the National Institute of Health Development approved this study (decision no 1001,issued 29 December 2021).

Funding

The authors received no financial support for the research,authorship,and/or publication of this article.

Guarantor

ORCID iD

Jekaterina Šteinmiller

Supplemental Material

Supplemental material for this article is available online.

References

Frisinger

Papachristou

. The voice of healthcare: Introducing digital decision support systems into clinical practice – a qualitative study. BMC Prim Care 2023; 24: 67.

Meunier

Raynaud

Guimaraes

, et al. Barriers and facilitators to the use of clinical decision support systems in primary care: A mixed-methods systematic review. Ann Fam Med 2023; 21: 57–69.

Sperl-Hillen

Rossom

Kharbanda

, et al. Priorities wizard: Multisite web-based primary care clinical decision support improved chronic care outcomes with high use rates and high clinician satisfaction rates. EGEMS 2019; 7: 9.

Febretti

Stifter

Keenan

, et al. Evaluating a clinical decision support interface for End-of-life nurse care. Ext Abstr Hum Factors Comput Syst 2014; 2014: 1633–1638.

Müller-Staub

Paans

. A standard for nursing process-clinical decision support systems (NP-CDSS). Stud Health Technol Inform 2016; 225: 810–811.

Casey

Reed

LeMaster

, et al. Physicians’ perceptions of clinical decision support to treat patients with heart failure in the ED. JAMA Netw Open 2023; 6: e2344393.

Pan

Wang

. Nursing decision support system: Application in electronic health records. Front Nurs 2020; 7: 185–190.

Conway

Adamson

Cunningham

, et al. Decision support for diabetes in Scotland: Implementation and evaluation of a clinical decision support system. J Diabetes Sci Technol 2018; 12: 381–383.

Sutton

Pincock

Baumgart

, et al. An overview of clinical decision support systems: Benefits, risks, and strategies for success. npj Digit Med 2020; 3: 17.

10.

Nilsson

Fagerström

. Decision-makers and mediators in a home healthcare digitisation process: Nurses’ experiences of implementation and use of a decision support system. Contemp Nurse 2018; 54: 511–521.

11.

Horwood

Luthuli

Mapumulo

, et al. Challenges of using e-health technologies to support clinical care in rural Africa: A longitudinal mixed methods study exploring primary health care nurses’ experiences of using an electronic clinical decision support system (CDSS) in South Africa. BMC Health Serv Res 2023; 23: 30.

12.

Jia

Zhang

Chen

, et al. The effects of clinical decision support systems on medication safety: An overview. PLoS One 2016; 15: e0167683.

13.

Marcolino

Oliveira

JAQ

Cimini

CCR

, et al. Development and implementation of a decision support system to improve control of hypertension and diabetes in a resource-constrained area in Brazil: Mixed methods study. J Med Internet Res 2021; 11: e18872.

14.

Abdulaal

Patel

Al-Hindawi

, et al. Clinical utility and functionality of an artificial intelligence-based app to predict mortality in COVID-19: Mixed methods analysis. JMIR Form Res 2021; 28: e27992.

15.

Silveira

Marcolino

Machado

, et al. Development and evaluation of a mobile decision support system for hypertension management in the primary care setting in Brazil: Mixed-methods field study on usability, feasibility, and utility. JMIR Mhealth Uhealth 2019; 7: e9869.

16.

Koskela

Sandström

Mäkinen

, et al. User perspectives on an electronic decision-support tool performing comprehensive medication reviews – a focus group study with physicians and nurses. BMC Med Inform Decis Mak 2016; 16: 6.

17.

Mebrahtu

Skyrme

Randell

, et al. Effects of computerised clinical decision support systems (CDSS) on nursing and allied health professional performance and patient outcomes: A systematic review of experimental and observational studies. BMJ Open 2021; 15: e053886.

18.

European Commission, https://ec.europa.eu/digital-building-blocks/wikis/pages/viewpage.action?pageId=533365863 (2023, accessed 12 December 2023).

19.

Heselmans

Delvaux

Laenen

, et al. Computerized clinical decision support system for diabetes in primary care does not improve quality of care: A cluster-randomized controlled trial. Implement Sci 2020; 15: 5.

20.

EBMEDS user’s guide to interpreting decision support, https://www.ebmeds.org/wp-content/uploads/sites/16/2021/04/Interpreting-decision-support-1.pdf (2023, accessed 1 October 2023).

21.

EBMEDS decision support. https://www.ebmeds.org/en/ (2023, accessed 1 November 2023).

22.

Ministry of Social Affairs, Tallinn University of Technology. Feasibility study for the development of digital decision support systems for personalised medicine, https://taltech.ee/en/emed-lab (2015, accessed 1 November 2023).

23.

Health Board, https://medre.tehik.ee/search/employees (2023, accessed 1 October 2023).

24.

Estonian Health Insurance Fund, https://www.tervisekassa.ee/en/partner/primary-health-care-quality-system (2023, accessed 11 December 2023).

25.

Gray

Grove

Sutherland

. Burns and Grove’s: The practice of nursing research, appraisal, synthesis, and generation of evidence. 8th ed. St. Louis: Elsevier, 2017.

26.

Sadler

Lee

Lim

RSH

, et al. Research article: Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy. Nurs Health Sci 2010; 12: 369–374.

27.

Baysari

Stocker

, et al. Researchers’ views on, and experiences with, the requirement to obtain informed consent in research involving human participants: A qualitative study. BMC Med Ethics 2020; 21: 93.

28.

Mets

Parsovs

. Time of signing in the Estonian digital signature scheme. Digit Evid Electron Signat Law Rev 2019; 16: 40–50.

29.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

30.

Braun

Clarke

. Reflecting on reflexive thematic analysis. Qual Res Sport Exerc Health 2019; 11: 589–597.

31.

Braun

Clarke

. Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning. Psychol 2013; 26: 120–123.

32.

Kortteisto

Kaila

Kunnamo

. Self-reported use and clinical usefulness of second-generation decision support – a survey at the pilot sites for evidence-based medicine electronic decision support (EBMeDS). Fin J eH eW 2009; 1: 161–169.

33.

Saputra

Ali

. Factors affecting decision support system: Knowledge, training, ease of use. Int J Bus Manag Adm 2023; 31: 1053–1058.

34.

Varonen

Kortteisto

Kaila

. What may help or hinder the implementation of computerized decision support systems (CDSSs): A focus group study with physicians. Fam Pract 2008; 25: 162–167.

35.

Rosen

DiazGranados

Dietz

. Teamwork in healthcare: Key discoveries enabling safer, high-quality care. Am Psychol 2018; 73: 433–450.

36.

Akbar

Lyell

Magrabi

. Automation in nursing decision support systems: A systematic review of effects on decision making, care delivery, and patient outcomes. J Am Med Inform Assoc 2021; 28: 2502–2513.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.10 MB

0.00 MB