Sage Journals: Discover world-class research

Abstract

Objective

There is an urgent need to prepare the healthcare workforce for the implementation of artificial intelligence (AI) into the healthcare setting. Insights into workforce perception of AI could identify potential challenges that an organisation may face when implementing this new technology. The aim of this study was to psychometrically evaluate and pilot the Shinners Artificial Intelligence Perception (SHAIP) questionnaire that is designed to explore healthcare professionals’ perceptions of AI. Instrument validation was achieved through a cross-sectional study of healthcare professionals (n = 252) from a regional health district in Australia.

Methods and Results

Exploratory factor analysis was conducted and analysis yielded a two-factor solution consisting of 10 items and explained 51.7% of the total variance. Factor one represented perceptions of ‘Professional impact of AI’ (α = .832) and Factor two represented ‘Preparedness for AI’ (α = .632). An analysis of variance indicated that ‘use of AI’ had a significant effect on healthcare professionals’ perceptions of both factors. ‘Discipline’ had a significant effect on Allied Health professionals’ perception of Factor one and low mean scale score across all disciplines suggests that all disciplines perceive that they are not prepared for AI.

Conclusions

The results of this study provide preliminary support for the SHAIP tool and a two-factor solution that measures healthcare professionals’ perceptions of AI. Further testing is needed to establish the reliability or re-modelling of Factor 2 and the overall performance of the SHAIP tool as a global instrument.

Keywords

artificial intelligence health informatics healthcare perception

Introduction

Artificial intelligence (AI) is the next wave of digital technology innovation impacting the healthcare setting as part of the fourth industrial revolution¹ or Health 4.0.² In the last decade, the advancements of AI in healthcare have been widely published.^3,4 However there is limited research investigating the experiences or perceptions of the healthcare workforce.⁵

In the past, technology implementation has provided considerable benefits to clinicians, but has also been shown to cause changes in cognitive structures, social interactions and clinical workflow efficiencies which creates unintended risk.^6–8 Qualitative studies show that healthcare professionals are enthusiastic to learn about AI and can see value in its capacity to augment their work.^9,10 However, as many healthcare professionals have had little exposure to AI, they are often concerned about the ethical implications of AI, the management of data, the disruption of the patient-physician relationship, and the development of professional knowledge.^4,9 Healthcare professionals primarily agree that AI could support the health and well-being of their patients, but suspect it will have organisational and professional impacts that they are not prepared for.^11,12 There are concerns that these perceptions could undermine the potential benefits of AI before they have been implemented, and at great cost to the organisation.

Tools that explore human-technology interaction tend to focus on the technology and serve the needs of the system designer during implementation, rather than identify the needs of the user. The Technology Acceptance Model (TAM)¹³ is a well-known tool that has explored the relationship between a user's perception and behaviour towards technology, and their likelihood to adopt it. The Technology Readiness Index (TRI)¹⁴ and more recently, the General Attitudes towards Artificial Intelligence Scale (GAAIS),¹⁵ measure the characteristics of an individual, such as ‘innovativeness’, ‘optimism’, ‘discomfort’ and ‘insecurity’, that may impact technology acceptance. However, they do not explore the baseline perception that healthcare professionals hold before they have begun using AI technology. Moreover, the current tools do not identify healthcare professional's feelings of preparedness, or their ideas about its impact to their professional role.

In 2019, we designed the SHAIP questionnaire which, unlike other human-technology interaction tools, focused solely on the healthcare professional and their perception of AI.¹⁶ The SHAIP questionnaire was the result of an Australian e-Delphi study, which gathered opinions of an interdisciplinary panel of experts in health and technology, geographically located in four Australian states. Five healthcare professionals from optometry, medicine, nursing and allied health, and three information technology experts from health technology, engineering and finance participated in the study. They brought a diverse range of experience in their disciplines, working as clinicians, business owners, academics and executive level managers. The SHAIP tool was underpinned by socio-technical systems theory^17–19 which acknowledges the complex relationships between the individual, the technology, and the workplace, and supports the belief that organisations need a human-centred understanding of workforce characteristics and perceptions when implementing new technology.^19–21 The e-Delphi design consolidated the diverse opinions of individual members of the panel into a single representative opinion. It culminated in an 11-item tool designed to capture healthcare professionals’ perceptions of AI in relation to their professional role, the extent to which they felt prepared for AI, and their views on its impact on clinical decision making and patient care. The aim of this study was to psychometrically evaluate and pilot the SHAIP questionnaire.

Methods

Procedures

A cross-sectional study was conducted in a regional health district in Australia. This district was chosen because it included primary, secondary and tertiary healthcare settings with a diverse range of healthcare and administrative professionals. Approximately 3000 employees who were over 18 years of age were invited to participate in the study. Job roles were classified as either ‘non-clinical roles’ such as administration, or ‘clinical roles’ like nursing/midwifery, medicine, allied health professionals, and other healthcare professionals. Allied health professions include speech pathology, podiatry, physiotherapy, social work and occupational therapy. A 5-point Likert scale: 1 = Strongly Disagree 5 = Strongly Agree was used to measure the participants’ perception of the 11 items.

The SHAIP questionnaire was uploaded into Qualtrics,²² a web-based survey tool. An invitation to participate in the study was sent to the staff of a local health district in regional New South Wales, Australia, via an email which contained a link to the SHAIP questionnaire and a detailed information sheet. An announcement regarding the study was also posted on the district's intranet. The questionnaire advised potential participants that consent was implied by completing the survey. The study was conducted according to the National Statement on Ethical Conduct in Human Research (2007)²³ and approved by the Southern Cross University Human Ethics Committee (HREC Register Number ECN-19-177).

Data analysis

Completed responses were imported into SPSS V.25²⁴ for analysis. Basic demographics were collected and participants were asked about their job description (including health employee, senior manager, educator), health discipline (including allied health, medicine, nursing/midwifery, other health and non-clinical) and current use of AI (yes, no or I don’t know). Exploratory factor analysis was conducted using principal axis factoring to understand how the SHAIP items group together to create a construct. The Chi square goodness of fit test, and Reproduced Correlation Matrix were evaluated to determine how well the factor represent the data. Cronbach's alpha was used to determine the reliability of each of the factors and Oblique Oblimin rotation which permit the factors to be correlated with one another, was used to interpret them. Independent variables (use of AI, discipline and job description) were compared with healthcare professionals’ measures of perception, using a one-way ANOVA.

Results

Descriptive analysis

A total of 252 healthcare professionals were recruited from August to November 2019. The 252 total responses provided more than 10 respondents per item and were therefore adequate for factor analysis (see Table 1).^11,25 More than half of the respondents were females (n = 159, 63.1%), aged between 41–60 years (n = 150, 59.6%). Participants represented ‘non-clinical roles’ (n = 24, 9.5%), and ‘clinical roles’, including nursing/midwifery (n = 141, 56%), medicine (n = 34, 13.5%), allied health professionals (n = 26, 10.3%), and other healthcare professionals (n = 27, 10.7%). Most participants were working as clinicians (n = 190, 75.4%) and held jobs that included health practitioner (n = 153, 60.7%), senior manager (n = 68, 27%) or educator (n = 31, 12.3%). Half of the respondents reported currently using AI (n = 126, 50%) (See Table 1).

Table 1.

Descriptive statistics.

Demographics n = 252		Total Sample
		n	%
Age		<20	.4
		21–30	9.1
		31–40	17.1
		41–50	28.6
		51–60	31.0
		61–70	12.7
		70 +	1.2
		Total	252
Gender	Male	90	35.7
	Female	159	63.1
	Not identified	3	1.2
Discipline	non-clinical staff	24	9.5
	nursing/midwifery	141	56.0
	medicine	34	13.5
	allied health	26	10.3
	other health	27	10.7
Clinician	Yes	190	75.4
	No	62	24.6
Job Description	Health practitioner	153	60.7
	Senior management	68	27.0
	Educator	31	12.3
Using AI	Yes	126	50.0
	No	105	41.7
	I don't know	21	8.3

Factor analysis and reliability

The Kaiser-Meyer-Olkin (KMO), a measure of sampling adequacy, was satisfactory at .804 and the result of the Chi Square for Bartlett's test (χ² = 758.567 and p < .001) suggested that patterned relationships can be found amongst the variables and the correlation matrix was satisfactory for factor analysis. The model was found to be a good fit with a Reproduced Correlation Matrix of 14%, and absolute values >.05.¹¹ Three factors were initially extracted but the third factor had a low eigenvalue (1.1), and a scree plot suggested that a two-factor solution was preferable. Item 11 had a very low communality (.115), did not load on any factor, and was removed. A subsequent factor analysis was re-run, extracting two factors which are shown in Table 2. Each item only loaded on one factor and the correlation between the two factors was .34. On the basis of the items that loaded on each factor, Factor 1 was interpreted to represent healthcare professionals’ perceptions of the Professional impact of AI, including improvements to care and decision making, as well as direct changes to their role. Factor 2 was interpreted to represent healthcare professionals’ perceptions of their Preparedness for AI, including adequate training, knowledge of ethical frameworks and readiness for its introduction. The final two factors accounted for 51.74% of the total variance. A cut-off for statistical significance of the factor loadings of 0.5 was considered,¹² however based on a sample size of 252, it was decided that factor loading would need to be at least .32 to be considered statistically meaningful.²⁶ Item 10 fell into the lower range (.362) and also had a low communality (.266): ‘I believe that should AI technology make an error; full responsibility lies with the healthcare professional’ . This item appears to have a legal or ethical angle which although an important indication of whether a healthcare professional has prepared themselves for the outcome of an AI-driven decision in healthcare, is a weaker item.

Table 2.

Factor loading of items in the SHAIP tool.

AI-IQ Items	Factor 1 Perception of professional impact	Factor 2 Perception of Preparedness for AI	Mean	SD
Item 1. I believe that the use of AI in my specialty could improve the delivery of patient care	.765		3.38	1.039
Item 2. I believe that the use of AI in my specialty could improve clinical decision making	.770		3.44	1.026
Item 3. I believe that AI can improve population health outcomes	.702		3.66	.866
Item 4. I believe that AI will change my role as a healthcare professional in the future	.655		3.74	.970
Item 5. I believe that the introduction of AI will reduce financial cost associated with my role	.452		2.99	.890
Item 6. I believe that overall healthcare professionals are prepared for the introduction of AI technology		.579	2.32	1.031
Item 7. I believe that one day AI may take over part of my role as a healthcare professional	.428		3.00	1.097
Item 8. I believe that I have been adequately trained to use AI that is specific to my role.		.576	2.25	1.058
Item 9. I believe there is an ethical framework in place for the use of AI technology in my workplace		.672	2.66	.903
Item 10. I believe that should AI technology make an error; full responsibility lies with the healthcare professional		.362	2.79	1.138
Total variance %	35.821	15.925
Mean of mean Scale Score	3.37	2.67

The individual raw scores for items 1 to 6 and 7 to 10 for each of the 252 participants were averaged and the overall mean scale scores were obtained for Professional impact of AI (3.66) and Preparedness for AI (2.55). In a preliminary analysis we examined the Cronbach's alpha estimate of internal consistency of each factor. Support for the 2-factor model required an alpha score = 0.70 or above.²⁷ The reliability estimate of Factor 1 Professional impact of AI was a Cronbach's alpha of .804 which suggests that items had very little variance specific to individual items. Factor 2 Preparedness for AI had a Cronbach's alpha score of .620 which is marginally less than our target.

Analysis of variance

A factorial between group analysis of variance (ANOVA) was used to investigate the effects of demographic variables such as age, use of AI, discipline and job description on perceptions of Professional impact of AI and Preparedness for AI.

Age:

There was no significant main effect of Age on perceptions of Professional impact of AI, F(6, 245 = .730, p = .636), or Preparedness for AI F(6, 245 = .739, p = .619). The majority of the healthcare professionals were over 40, however this is likely an accurate representation of the current healthcare professional demography.

Use of AI:

There was a significant main effect of healthcare professionals’ Use of AI (yes/no/I don’t know) on perception of Professional impact of AI, F(2, 249 = 10.45, P < .001). A Tukey's test of multiple comparisons (P < .05) showed that the mean scale score for ‘Yes’ was significantly higher than the mean scale score for the other two categories: ‘No’ and ‘I don’t know’ (see Table 3). There was also a significant main effect of Use of AI on perception of Preparedness for AI F(2, 249 = 8.78, P < .001). The Tukey test showed that the mean scale score for ‘Yes’ was significantly higher than the mean scale score for the ‘No’ category. These results suggest that healthcare professionals who use AI were unsure or agreed (mean scale score = 3.55) that AI will change their role as a healthcare professional and that on average (mean scale score = 2.82) disagree that they are prepared for its use in the delivery of care than those that are not using it.

Table 3.

SHAIP tool subscale and total summary scores [mean scale score (SD)].

Subscale Total (n = 252)	Professional impact of AI	p-value	Preparedness for AI	p-value
Age
<20 (n = 1)	4.5		2.25
21–30 (n = 23)	3.38 (1.04)		2.41 (0.82)
31–40 (n = 43)	3.46 (0.78)		2.48 (0.69)
41–50 (n = 72)	3.28 (0.65)		2.44 (0.77)
51–60 (n = 78)	3.36 (0.63)		2.52 (0.65)
61–70 (n = 32)	3.36 (0.57)		2.69 (0.62)
70 + (n = 3)	3.50 (0.44)		2.58 (0.52
Use of AI
Yes (n = 126)	3.55 (0.61)*	<.001	2.82 (0.66)*	<.001
No (n = 105)	3.21 (0.74)		2.53 (0.55)
I don’t know (n = 21)	3.01 (0.69)		2.50 (0.62)
Discipline
Non-clinical staff (n = 24)	3.25 (0.67)		2.92 (0.67)
Nursing/midwifery (n = 141)	3.36 (0.68)		2.69 (0.62)
Medicine (n = 34)	3.59 (0.68)		2.59 (0.55)
Allied Health (n = 26)	2.94 (0.82) *	<.002	2.38 (0.60)
Other Health (n = 27)	3.60 (0.55)		2.77 (0.64)
Job Description
Health Practitioner (n = 153)	3.30 (0.69)		2.70 (0.60)
Senior Management (n = 68)	3.40 (0.77)		2.70 (0.67)
Educator (n = 31)	3.56 (0.54)		2.52 (0.64)

*Indicates a p value of <0.05. This item is significantly different from the other items in its category.

Discipline:

There was a significant main effect of Discipline (nursing/midwifery/allied health/medicine/other health/non-clinical) on perceptions of Professional impact of AI, F(4, 247 = 4.41, P < .002). A Tukey's test of multiple comparisons (P < .05) showed that allied health mean scale score was significantly lower than all other disciplines (see Table 3). These results suggest that allied health professionals on average disagree or are unsure (mean scale score = 2.94) whether AI will impact their role compared to other healthcare professionals. Allied health professionals’ representation (n = 26, 10.3%) appears underpowered, compared to nurses/midwifery (n = 141, 56%), medicine (n = 34, 13.5%), and other healthcare professionals (n = 27, 10.7%). although may reflect the ratio of allied health professionals working in the acute care settings. There was no significant main effect of Discipline on perceptions of Preparedness for AI F(4, 247 = .552, p = .580). Low mean scale score across all disciplines suggests that all disciplines perceive that they are not prepared for AI.

Job description:

There was no significant main effect for Job Descriptions on perceptions of Professional impact of AI F(2, 249 = .953, p = .146) or Preparedness for AI F(2, 249 = 1.763, p = .040) (see Table 3). Mean scale scores across all disciplines were close to the neutral point of 3 (unsure) so a one-sample t-test was conducted. The t-test found that overall Professional impact of AI t(251) = 8.307, p < .001 and Factor 2 ‘Preparedness for AI’ t(251) = 8.750, p < .001, participants tend to be more likely to respond positively to items about the professional impact of AI on job description and tend to perceive themselves as less than prepared for AI than would be expected if they were responding as ’unsure’.

Discussion

Findings support the structure and reliability of a 10-item SHAIP questionnaire. Patterned relationships were found amongst the items (KMO = .804 and Chi Square for Bartlett's test (χ² = 758.567 and p < .001), and the correlation matrix was satisfactory for factor analysis. The model was found to be a good fit when Item 11 was removed due to a low communality (.115), because only a small proportion of its variance could be explained by the factors. The 10 items used in final analysis, loaded on to 2 factors: Factor 1 Professional impact of AI and Factor 2 Preparedness for AI. The reliability estimate of Factor 1 (α = .804) conformed to Cronbach's definition of equivalence²⁸ and items had very little variance specific to individual items. Factor 2 Preparedness for AI (α = .620) did not meet this target suggesting it is multidimensional in nature and may require other items that have not been elucidated by the original e-Delphi study. A review of this factor and how it can incorporate more global views is recommended to improve its application.

The SHAIP tool is different than other human-technology interaction tools, in that it acts as a baseline measurement of healthcare professionals’ perception of AI in relation to its ‘impact to their professional role’, and their ‘preparedness’. The GAAIS tool,¹⁵ published in 2020 after this study was conducted, is not specific to healthcare but found that ‘comfortableness’ was a strong predictor of attitude towards AI. The authors found that AI applications related to health were amongst the lowest rated for both ‘comfortableness’ and ‘perceived capability’, and that a focus on the complex domain of healthcare was needed. It is suggested in the literature that healthcare professionals and possibly healthcare organisations, are not clear on what AI is.²⁹ That being true, if a healthcare professional does not understand what AI is or how it is applied to the delivery of care, they are not able to associate a negative or positive perception to these questions. The SHAIP tool is a conduit for organisations to begin a conversation about AI.

While we offer some conceptual insights into healthcare professionals’ perceptions of AI, some study limitations are acknowledged. First, the initial e-Delphi framing was conducted by a panel of Australian interdisciplinary experts in health and technology which may limit the tool's use as a global instrument. Secondly, while the study analysed 252 complete surveys, this only represents a small proportion of the population of healthcare professionals working in the regional NSW local health district. It is unclear if socio-economic factors such as the geographical location of this survey impacted the results and whether there may be a greater exposure to AI in metropolitan healthcare settings, which is a consideration for further work. The study is also at high risk of non-response bias, which may impact the generalisability of the findings.

Conceptual insights

There was no significant main effect of Age on perceptions of Professional impact of AI or Preparedness for AI, but the study found that ‘use of AI’ did influence the perception of both factors. Due to the neutral framing of the items, where a perception would ordinarily be expected to be highly divergent based on age, profession or experience, the tool instead measures the magnitude with which healthcare professionals perceive a statement to be true. Interestingly, healthcare professionals who used AI strongly agreed that it would impact their role, but also felt more prepared for its use than healthcare professionals who were not using AI. Technology use has been shown to improve an individual's perception of competence, allowing them to see the merits of the technology more easily.^30–33 However, without an understanding of how it will enhance performance and improve care delivery, technology use may fail to reveal its benefits and lead to misuse or abandonment by the workforce.³⁴ Sheng et al.³⁵ found that physicians who were given hands-on experience of new telemedicine technology before its implementation and were supported with training, were able to single out the potential value of the technology in respect to their role and expressed a great interest in learning more about it. This result echoes the call for inclusive design,³⁶ workforce training and development, and an ethical framework for artificially intelligent healthcare systems to realise the potential value this technology could offer.

‘Discipline’ had a significant main effect on Allied healthcare professionals’ perceptions of Factor 1, strongly disagreeing that AI would have a professional impact on them, which differed from other health disciplines and warrants further exploration. The results showed that the study was heavily skewed in favour of nurses, and the representation of doctors as a key user of AI technology in the future was also low. The scope and versatility of AI is expected to impact all stakeholders in the healthcare workforce^6,37 and the implications for AI technology implementation at a personal and organisational level may be profound. In Australia, technologies such as telehealth have been met by resistance from some members of the allied health community.⁷ An Australian government survey exploring allied health professionals’ preparedness for e-Health technology found that the perception of its benefits was mixed. There was inequity across the allied health network to accommodate the technology and a lack of understanding about its risks and benefits that resulted in isolated technology adoption.⁸ The results of this pilot suggests that some healthcare professionals in Australia are unaware of the nature and extent of AI and its planned applications in healthcare. A coordinated effort will be required from professional bodies to include, educate and orient the workforce to this type of technology, and at the same time develop a framework that addresses the human-centred challenges and diverse needs of the healthcare community.^8,38,39 This framework should prioritise an inclusive design element to any new AI technology application, encourage an interdisciplinary approach to implementation and ensure that education is initiated early and is targeted to the needs of the discipline.

The promise that AI innovation will improve care delivery and reduce workload is attractive to organisations but there is doubt amongst healthcare organisations that the implementation of technology such as AI is viable, because of the investment that is required for infrastructure, training and policy development.^40–42 The socio-technical systems theory recommends that the individual, the organisation and the technology are given equal consideration before the introduction of new technology.⁴³ To help organisations maximise the benefit of this new technology, it is important to explore the experiences and perceptions of the healthcare workforce, so that barriers to implementing the technology can be identified, and training and education can be targeted to their needs.^44,45

Conclusions

The results of this study provide preliminary support for the SHAIP tool and a two-factor solution that measures healthcare professionals’ perceptions of AI. Further testing will establish the reliability or re-modelling of Factor 2 specifically. The tool is designed to measure healthcare professionals’ perceptions of AI, as a baseline for organisations.

Footnotes

Acknowledgements

None

Author Contributions

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) received no financial support for the research,authorship and/or publication of this article.

Ethical approval

This study protocol was conducted according to the national statement on Ethical Conduct in Human Research (2007) and approved by the Southern Cross University Human Ethics Committee (HREC Register Number ECN-19-177).

Guarantor

Contributorship

LS,SG,CA and SS conceived of the presented idea. AS contributed to the acquisition of the data. LS carried out the research and analysis with the support of SG,CA,SS and AS. LS wrote the manuscript and SG,CA,SS and AS contributed to the drafting and revising of the manuscript.

Informed Consent

Not applicable,because this article does not contain any studies with human or animal subjects.

Trial Registration

Not applicable,because this article does not contain any clinical trials.

ORCID iD

Lucy Shinners

References

Schwab

. The Fourth Industrial Revolution: what it means, how to respond 2016 [Available from: https://www.weforum.org/agenda/2016/01/the-fourth-industrial-revolution-what-it-means-and-how-to-respond/.

Thuemmler

Bai

. (Eds.). Health 4.0: how virtualization and big data are revolutionizing healthcare. (pp. 2168–2194). Cham Switzerland: Springer International Publishing, 2017.

Esteva

Robicquet

Ramsundar

, et al. A guide to deep learning in healthcare. Nat Med 2019; 25(1): 24–29.

Collado-Mesa

Alvarez

Arheart

. The role of artificial intelligence in diagnostic radiology: a survey at a single radiology residency training program. J Am Coll Radiol 2018; 15: 1753–1757.

Shinners

Aggar

Grace

, et al. Exploring healthcare professionals’ understanding and experiences of artificial intelligence technology use in the delivery of healthcare: an integrative review. Health Informatics J 2020; 26: 1225–1236.

Baxter

, et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019; 25: 30–36.

Speyer

Denman

Wilkes-Gillan

, et al. Effects of telehealth by allied health professionals and nurses in rural and remote areas: a systematic review and meta-analysis. J Rehabil Med 2017; 50(3): 225–235.

Australian Goverment. The eHealth Readiness of Australia’s Allied Health Sector. In: Department of Health and Ageing, editor. Commonwealth of Australia; 2011.

Laï

Brian

Mamzer

. Perceptions of artificial intelligence in healthcare: findings from a qualitative survey study among actors in France. J Transl Med 2020; 18: 14.

10.

Paranjape

Schinkel

Nannan Panday

, et al. Introducing artificial intelligence training in medical education. JMIR Med Educ 2019; 5: e16048.

11.

Comrey

Lee

. A first course in factor analysis. Psychology press, 2013.

12.

Hair

. Multivariate data analysis. 2009.

13.

Davis

. Percieved usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989; 13: 319–340.

14.

Lam

Chiang

Parasuraman

. The effects of the dimensions of technology readiness on technology acceptance: an empirical analysis. J Interact Mark 2008; 22: 19–39.

15.

Schepman

Rodway

. Initial validation of the general attitudes towards Artificial Intelligence Scale. Comput Hum Behav Rep 2020; 1: 100014.

16.

Shinners

Aggar

Grace

, et al. Exploring healthcare professionals’ perceptions of artificial intelligence: Validating a questionnaire using the e-Delphi method. Manuscript accepted for publication. Digital Health. 2020.

17.

Esteva

Kuprel

Novoa

, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017; 542: 115–118.

18.

Doherty

King

. An investigation of the factors affecting the successful treatment of organisational issues in systems development projects. Eur J Inf Syst 2001; 10: 147–160.

19.

Appelbaum

. Socio-technical systems theory: an intervention strategy for organizational development. Manage Decis 1997; 35: 452–463.

20.

Adaba

Kebebew

. Improving a health information system for real-time data entries: an action research project using socio-technical systems theory. Inform Health Soc Care 2018; 43: 159–171.

21.

Clark

Dean

Bolton

, et al. Bench to bedside: the technology adoption pathway in healthcare. Health Technol (Berl) 2020; 10: 537–545.

22.

Qualtrics. Available from: https://www.qualtrics.com.

23.

Anderson

. 2007 national statement on ethical conduct in human research. Intern Med J 2011; 41: 581.

24.

Corp

. IBM SPSS statistics for windows. Version 25.0 ed. Armonk, NY: IBM Copr., 2017.

25.

Waltz

Strickland

Lenz

. Measurement in nursing and health research. Springer Publishing Company, 2010.

26.

Tabachnick

Fidell

Ullman

. Using multivariate statistics. Boston, MA: Pearson, 2007.

27.

Byrne

. Structural equation modeling with lisrel, prelis, and simplis: basic concepts, applications, and programming. 1st ed. Psychology Press, 1998.

28.

Cronbach

. Coefficient alpha and the internal structure of tests. Psychometrika 1951; 16: 297–334.

29.

Johnson

Verdicchio

. Reframing AI discourse. Minds Mach 2017; 27: 575–590.

30.

Zayyad

Toycan

. Factors affecting sustainable adoption of e-health technology in developing countries: an exploratory survey of Nigerian hospitals from the perspective of healthcare professionals. Peerj 2018; 6: e4436.

31.

Venkatesh

Moris

Davis

, et al. User acceptance of information technology: toward a unified view. MIS Q 2003; 27: 425–478.

32.

Homes

Nugent

Augusto

. Human-robot user studies in eldercare: lessons learned. Smart homes and beyond: Icost, 2006; 4.

33.

Gessl

Schlögl

Mevenkamp

. On the perceptions and acceptance of artificially intelligent robotics and the psychology of the future elderly. Behav Inf Technol 2019; 38: 1068–1087.

34.

Nemeth

Feifer

Stuart

, et al. Implementing change in primary care practices using electronic medical records: a conceptual framework. Implement Sci 2008; 3: 3.

35.

Liu Sheng

Jen-Hwa Hu

Wei

C-P

, et al. Adoption and diffusion of telemedicine technology in health care organizations: a comparative case study in Hong Kong. J Org Comput Electron Commer 1998; 8: 247–275.

36.

OECD. Health in the 21st Century 2019.

37.

Topol

. Preparing the healthcare workforce to deliver the digital future. Englande HE: United Kingdom National Health Service, 2019.

38.

Bradford

Hurdle

Lasalle

, et al. Development of a HIPAA-compliant environment for translational research data and analytics. J Am Med Inform Assoc 2014; 21: 185–189.

39.

Iacono

Stagg

Pearce

. A scoping review of Australian allied health research in ehealth. BMC Health Serv Res 2016; 16.

40.

Morren

. Technology paradoxes in the practice of healthcare. Enschede, The Netherlands: University of Twente, 2019.

41.

O'Connor

Bowles

. Telehealth and mHealth. Res Nurs Health 2021; 44: 3–4.

42.

Lewinski

Rushton

Van Voorhees

, et al. Implementing remote triage in large health systems: a qualitative evidence synthesis. Res Nurs Health 2021; 44: 138–154.

43.

Sittig

Singh

. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19: i68–i74.

44.

Stevenson

Nilsson

Petersson

, et al. Nurses’ experience of using electronic patient records in everyday practice in acute/inpatient ward settings: a literature review. Health Informatics J 2010; 16: 63–72.

45.

Rogers

. Diffusion of innovations. New York; London: Free Press; Collier Macmillan, 1983.