Sage Journals: Discover world-class research

Abstract

Background

As the field of robotics and smart wearables continues to advance rapidly, the evaluation of their usability becomes paramount. Researchers may encounter difficulty in finding a suitable questionnaire for evaluating the usability of robotics and smart wearables. Therefore, the aim of this study is to identify the most commonly utilized questionnaires for assessing the usability of robots and smart wearables.

Methods

A comprehensive search of databases, including PubMed, Web of Science, and Scopus, was conducted for this scoping review. Two authors performed the selection of articles and data extraction using a 10-field data extraction form. In cases of disagreements, a third author was consulted to reach a consensus. The inclusions were English-language original research articles that utilized validated questionnaires to assess the usability of healthcare robots and smart wearables. The exclusions comprised review articles, non-English publications, studies not focused on usability, those assessing clinical outcomes, articles lacking questionnaire details, and those using non-validated or researcher-made questionnaires. Descriptive statistics methods (frequency and percentage), were employed to analyze the data.

Results

A total of 314 articles were obtained, and after eliminating irrelevant and duplicate articles, a final selection of 50 articles was included in this review. A total of 17 questionnaires were identified to evaluate the usability of robots and smart wearables, with 10 questionnaires specifically for wearables and 7 questionnaires for robots. The System Usability Scale (50%) and Post-Study System Usability Questionnaire (19.44%) were the predominant questionnaires utilized to assess the usability of smart wearables. Moreover, the most commonly used questionnaires for evaluating the usability of robots were the System Usability Scale (56.66%), User Experience Questionnaire (16.66%), and Quebec User Evaluation of Satisfaction with Assistive Technology (10%).

Conclusion

Commonly employed questionnaires serve as valuable tools in assessing the usability of robots and smart wearables, aiding in the refinement and optimization of these technologies for enhanced user experiences. By incorporating user feedback and insights, designers can strive towards creating more intuitive and effective robotic and wearable solutions.

Keywords

Evaluation assessment usability robots wearables

Introduction

The lack of usability in robots and smart wearables poses significant challenges that hinder their effective integration and adoption. Firstly, it hinders user adoption and acceptance, as complex interfaces and controls make these technologies difficult to operate for individuals without specialized knowledge or training. This leads to frustration and limited engagement with the devices.¹ Secondly, inadequate usability hinders the effortless assimilation of robots and wearables into daily routines, limiting the potential benefits and functionality of robots and wearables.² The lack of intuitive and natural interaction further restricts user engagement and hampers the overall user experience.³ Moreover, poor usability can result in errors, inefficiencies, and decreased productivity, undermining the intended advantages of these technologies.⁴ Overcoming these challenges is essential to unlock the full potential of robots and smart wearables, ensuring their effective utilization, and maximizing user satisfaction.⁵ One of the ways to overcome these challenges is to evaluate the continuous usability of these technologies.

Assessing and measuring the usability of robots and smart wearables are crucial steps in understanding how well these technologies meet user needs and expectations.^6,7 To accomplish this, researchers and practitioners rely on questionnaires specifically designed to evaluate the usability of these devices.⁶ These questionnaires serve as valuable tools for gathering user feedback, identifying areas for improvement, and enhancing the overall user experience. As far as we know, no study has been conducted to identify the most common usability evaluation questionnaires for robots and smart wearables. Only two studies have identified and introduced the most common questionnaires for evaluating telemedicine services,⁸ as well as assessing satisfaction, acceptance, usability, and quality outcomes of m-health Apps.⁹ Moreover, existing literature has diligently explored usability scores in digital health,¹⁰ the usability of robots and smart wearables has remained a relatively underexplored frontier. The challenges posed by the lack of intuitive interaction and integration into daily routines are distinctive to these technologies and necessitate a specialized focus.¹¹ Meyer et al.,⁶ highlighted the scarcity of dedicated studies evaluating the usability of robots and wearables, emphasizing the lack of clear evaluation tools and guidelines, despite the consensus on the importance of usability in user-centered design. Khakurel et al.,⁷ noted that due to the relatively recent emergence of wearable devices as a field of study, there is a scarcity of research examining usability and usability evaluation tools and its correlation with various types of wearable technology. Conducting various studies to introduce usability evaluation tools, evaluation methods, standards, and metrics can be very helpful in this field.

Therefore, this scoping review endeavors to provide a comprehensive understanding of the most utilized questionnaires for evaluating the usability of robots and smart wearables. By illuminating this specific facet, our study offers novel insights that bridge a critical gap in the existing literature. Our study distinguishes itself from previous work in several ways. First, we focus specifically on robots and smart wearables, which have distinct usability characteristics compared to other digital health technologies. Second, we adopt a comprehensive scoping review methodology, ensuring a thorough examination of the available literature. Third, we provide a detailed analysis of the identified questionnaires.

Moreover, this scoping review not only contributes to the ongoing discourse on usability evaluations but also serves as a valuable resource for researchers, practitioners, and developers working in robotics and wearable technology. By providing a comprehensive overview of the most commonly used questionnaires, we assist in the selection and application of appropriate tools for usability assessment. Moreover, our findings pave the way for future research on usability evaluation methodologies, ultimately contributing to the development of more intuitive, user-friendly, and impactful robots and smart wearables.

Material and methods

The Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist was utilized in this study to ensure the selection of studies and proper reporting of the results (More details in Appendix A).

Information sources and search strategy

To identify articles pertaining to questionnaires used for evaluating the usability of robots and smart wearables, a comprehensive search was conducted in three databases: PubMed, Web of Science, and Scopus, until May 30, 2023. To search these databases, relevant keywords related to robots, wearables, usability, and questionnaires were utilized. The following combinations of keywords were employed to locate relevant articles:

((Robots OR smart wearables OR wearable OR wearable devices OR wearable apparatus) AND (usability evaluation OR usability assessment) AND (questionnaire))

Inclusion and exclusion criteria

In order to ascertain the inclusion of studies, we systematically applied predefined inclusion and exclusion criteria, detailed in Table 1. These criteria served as the basis for our selection process, ensuring a rigorous and transparent approach to study inclusion.

Table 1.

Study inclusion and exclusion criteria summary.

Criteria type	Inclusion criteria	Exclusion criteria
Study design	Original observational and interventional research articles using a valid and referenced questionnaire	Review articles
Language	English language publications	Non-English language publications
Topic focus	Usability evaluation of robots and smart wearables in the healthcare field	Studies not specifically addressing the usability of robots and smart wearables
Outcome evaluation	Valid and referenced questionnaire for usability assessment	Studies evaluating clinical outcomes
Questionnaire details	Articles with details about the questionnaires used	Articles lacking details about the questionnaires used
Questionnaire status	Valid and referenced questionnaire for usability evaluation of robots and smart wearables	Questionnaires that have not undergone validation and researcher-made questionnaires

Paper selection

In the initial stage, abstracts of all relevant articles were obtained from three databases, namely PubMed, Web of Science, and Scopus, and were imported into the EndNote X9 software by the author, KHM. Duplicate articles were subsequently eliminated from the dataset. Then, two authors (KHM, RM) independently reviewed all the retrieved papers based on their titles and abstracts. Subsequently, the same individuals evaluated the full text of the selected papers. In cases where disagreements arose, the opinion of another author (KB) was sought.

Data extraction

The included articles were analyzed, and the following information was extracted: first author's name, year of publication, country of origin, study aim, target population, type of wearable, type of robot, sample size, sample size based on breakdown of male and female participants, intervention period, and the name of the questionnaire used.

Quality appraisal

Using the 2018 version of the Mixed Methods Appraisal Tool (MMAT), authors RM and KHM independently conducted a comprehensive assessment of the studies.¹² Any disagreements between the original author were resolved through discussions with an additional author (KB) until a consensus was reached. The evaluation of studies was carried out in accordance with the MMAT criteria relevant to the chosen category. It is noteworthy that the most recent version of MMAT provides a descriptive quality assessment rather than a cumulative numerical score. In each study category, response options included “yes,” “no,” and “can't tell”. The selection of “can't tell” signified insufficient information in the study to provide a definitive “yes” or “no” response.¹³

Synthesis of results

After storing and managing the data in MS Excel for processing, one author (KHM) thoroughly reviewed the imported data, conducting tasks such as spell-checking and cell formatting to ensure accuracy and consistency. Descriptive statistics, specifically frequency and percentage calculations, were employed to summarize the collected data. The descriptive data derived from the findings of the included articles were meticulously organized into tables and figures based on thematic categorization. This approach facilitated the presentation of the review's findings and guided the study aims by KB and RM.

Results

A total of 314 articles were initially retrieved for this study. After excluding duplicates, a careful review and assessment of the remaining 268 studies was conducted based on predefined inclusion and exclusion criteria. Ultimately, 50 articles met the criteria and were included in the study. The details of the search process and study selection are visually presented in Figure 1.

Figure 1.

Study selection process.

Characteristics of the included studies

A comprehensive summary of the chosen articles is outlined in Table 2.

Table 2.

Wearable devices
Ref	Country	Study aims	Target population	Wearable type	Sample size	Male	Female	Intervention period	Questionnaire name
¹⁴	Canada	Assessing the usability and wearability of a vibro-tactile display belt, designed to relay physiological information to anesthesiologists.	Anesthesiologists	Vibro-tactile display belt	17	12	5	75 min per case	PSSUQ
¹⁵	Chicago	Evaluation of the usability of a tongue drive system by people with high-level spinal cord injury	People with high-level spinal cord injury	TDS	11	9	2	6 weeks	PSSUQ
¹⁶	Portugal	Evaluating of a wearable system for home-based monitoring of Parkinson's symptoms	Patients with Parkinson	A wearable system for home-based assessment of Parkinson's symptoms	22	14	8	12 weeks	PSSUQ
¹⁷	Switzerland	Evaluation of the applicability and usability of a wearable accelerometer device for the diagnosis of bilateral tonic-clonic seizures	Patients with bilateral tonic-clonic seizures	Wearable accelerometer device	71	39	32	15 months	PSSUQ
¹⁸	Switzerland	Evaluation of adherence and acceptance of long-term use of a home-monitoring wearable by patients with Parkinson's disease	Patients with Parkinson's disease	Home-Monitoring wearable	4	2	2	1-month	SUS
¹⁹	UK	Evaluation of usability, feasibility and acceptability of a wrist-worn activity tracker relapse in major depressive disorder	Patient with depressive disorder	wrist-worn activity tracker	-	-	-	2-years	PSSUQ, and TAM-FF
²⁰	South Korea	Usability analysis of FRAS	older people	wearable FRAS	20	-	-	30 min	SUS, and MARS
²¹	Italy	Usability evaluation of a Smart Wearable for motion tracking for the upper limbs	Patients with Parkinson	Smart wearable for motion tracking for the upper limbs	25	15	10	1-h	PSSUQ
²²	Switzerland	Development And Evaluation Of A Wearable Skeleton For Wrist Extension/Flexion Support In Stroke Rehabilitation	Patients with stroke	Wrist exoskeleton	17	10	7	1-week	SUS, and NASA-Task Load Index (RTLX)
²³	Ireland	Comparing the usability of multiple wearable sensors by the older adult population	Older adults	Smart wearables to track physical activity	8	5	3	1-week	IMI, and SUS
²⁴	Netherlands	Identifying negative and positive effects as well as facilitators and barriers for the use of wearable device for monitoring of vital signs	Patients admitted to the internal medicine and surgical wards of the hospital	A wearable device for monitoring of vital signs	100	60	30	45 min	SUS
²⁵	UK	Evaluating the feasibility, usability, and acceptability of a wrist-worn device in remote assessment of disease and relapse in epilepsy	Patient with epilepsy	Wrist-worn device	-	-	-	6-months	PSSUQ
²⁶	Netherlands	Examining users’ experiences and health related effects of exoskeleton in people with complete spinal cord injury	Patients with spinal cord injury	Wearable exoskeleton	14	7	7	3-weeks	QUEST and SUS
²⁷	UK	Assessing the feasibility and acceptability of utilizing an activity tracker for self-management of physical activity in patients with COPD	Patients with COPD	Activity tracker	30	17	13	Twice weekly for 6 to 7 weeks	SUS
²⁸	Denmark	Evaluating a smart watch to assessment cognitive performance and check the usability of this tool	Patients with cognitive function disorders	Smartwatches	21	12	9	60–75 min per participant	MARS
²⁹	Denmark	This paper presents a study of two smartwatch-based cognitive tests to assess participants’ attention and working memory.	Patients with cognitive function disorders	Smartwatch	21	12	9	Two min per participant	MARS
³⁰	France	evaluation the user acceptability of a smart shoe insole by frail older adults	Older adults	Smart Shoe Insole	9	3	6	At least 5 min with a tolerance of 1 min	QUEST
³¹	UK	Developing and assessing the feasibility of a digital intervention based on ACT specifically designed for individuals diagnosed with GAD	Adults with a diagnosis of GAD	A paired wearable device is used to collect data on both physical activity and sleep continuity	10	0	0	2-weeks	SUS
³²	Netherlands	Examining the acceptability and usability of a wearable consisting of smartphone and smartwatch by 10 patients with aggressive behavior	Patients with aggressive behavior	A wearable consisting of smartphone and smartwatch	10	9	1	2-week	SUS
³³	Italy	Evacuating the upper limb functional improvement in muscular dystrophy patients through the use of both a passive and a semi-active device and comparing the impact of this technology on arm functionality	Patients with muscular dystrophy	Assistive system for arm functions improvement	36	32	4	2-weeks (4 h per day)	SUS
³⁴	Republic of Korea	Assessing the time-efficiency and feasibility of using a patchy-type 12-lead ECG measuring and transmitting device (P-ECG) by an EMT in an ambulance during patient transport	EMT	Patchy-type wireless 12-lead electrocardiogram	18	14	4	15 min of education on 12-lead ECG and the devices and 10 min of practice	SUS
³⁵	Republic of Korea	Design and evaluate an mHealth platform that includes a smartphone application synchronized with a wearable activity tracker and a web-based portal	Patients with peripheral artery disease	A wearable activity tracker	7	4	3	Seven days	SUS
³⁶	France	Investigating the technical feasibility of monitoring stretching exercises by a humanoid robot trainer for chronic back pain	Patients with CLBP	Humanoid Robot Coach for CLBP	27	-	-	4-weeks	SUS
³⁷	Italy	Design and evaluation of a wearable exoskeleton for rehabilitation of post-stroke persons	Patients with stroke	Wearable exoskeleton	5	5		1-day	SUS
³⁸	Spain	Assessing the safety, usability, personal experiences, user satisfaction, and effort expended following an exergame using a Wearable IVR device in individuals with PD	Patients with Parkinson	A wearable HMD to promote physical activity in patients with Parkinson's disease	32	25	7	Two sessions per person	SUS
³⁹	Germany	Examining the impact of a comprehensive self-regulation intervention on moderate to vigorous physical activity and step count, utilizing commercially available activity trackers and evaluating the acceptance and usability of these devices	Older adults	Activity tracker	80	33	47	21-days	TSQ-WT
⁴⁰	Portugal	(1) Evaluating the usability, acceptability, and user satisfaction with the Activity tracker 2 in older adults; (2) examining the mediating effect of the usability on the relationship between user specifications and satisfaction; and (3) exploreing the moderating effect of user characteristics on the relationship between usability and user satisfaction	Older adults	Activity tracker	110	50	60	2over 15 days	USEQ, and SUS
⁴¹	France	Assessing Usability, User Experience (UX), Patients’ Perceptions of Body Worn Sensors (BWS) and Health Care Providers’ Opinions on BWS Monitoring	Patients with Parkinson	Body-Worn Sensors (BWS) for continuous assessment of patients with Parkinson motor symptoms	22	9	13	A week (30–60 min)	SUS
⁴²	Italy	Evaluating usability of and satisfaction with Finger-size wearable assistive technology device for visually impaired	Visually impaired people	Finger-size wearable assistive technology device	100	51	49	6-months	SUS, and QUEST
Robots
Ref	Country	Study aims	Target Population	Robot type	Sample size	Male	Female	Intervention period	Questionnaire name
⁴³	Italy	Design and evaluation of an assistive device for upper limb support	People with severe neuro-motor disabilities	Robotic exoskeleton arm	6	5	1	A single day session	TSQ-WT, and SUS
⁴⁴	UK	Evaluation of human-robot interaction in a natural environment	-	Assistive robot to interact with humans	12	8	4	Once a week for a period of 10 weeks	SUS
⁴⁵	Netherlands	Evaluation of a rehabilitation robot for upper limb disabilities	Patient with upper extremity disability	Rehabilitation robot	5	4	1	6-weeks, 3 days per week, 60 min per day	SUS
⁴⁶	Spain	Design and evaluation of a robot-based tool for physical and cognitive rehabilitation	Older people	A robot-based tool for physical and cognitive rehabilitation	7	-	-	Once every 3 months (duration of the session: 25 min)	SUS
⁴⁷	Belgium	Investigating the motivation and expectations of stroke patients and therapists regarding the usability of robot-assisted walking training	Stroke patients	Robot-assisted gait rehabilitation	46	26	20	A week for approximately 30–45 min	Usefulness, Satisfaction and Ease of Use Questionnaire (USE)
⁴⁸	Germany	Evaluation of automated real-time feedback provided by an on-shoe inertial sensor-based gait system in people with gait disorders after incomplete spinal cord injury, stroke and in the elderly.	Patients with gait disorders	Shoe-mounted inertial-sensor-based gait therapy system	1860	919	941	4-weeks	QUEST
⁴⁹	Canada	Comparing the effectiveness of two feedback approaches, Scores + Visual + Force feedback and Visual + Force feedback, in reducing trunk compensation	Hemiparetic stroke survivors	Rehabilitation robotic arms	14	8	6	-	SUS
⁵⁰	Poland	Evaluation of the clinical applicability of a robotic assistant for patients with mild cognitive impairment and Alzheimer's	Patients suffering from MC) and AD	Robotic assistant for aging patients with memory impairments	18	6	12	-	UEQ
⁵¹	Germany	Comparing the usability, acceptability, and barriers of usage of a robot-supported gait rehabilitation system between younger group and an older group	Patients with gait impairments	Robot-supported gait rehabilitation system	13	11	2	Five-therapy sessions, each lasting 20 min	SUS
⁵²	Italy	Evaluation of usability, and acceptance of a robotic walker from the perspective of patients and clinical professionals	Patients with walking disorders	Robotic walker	42	20	22	6 months	SUS, and UEQ
⁵³	USA	Identification and determination of barriers related to the use of an exoskeleton to reduce musculoskeletal symptoms for workers in the operating room	Workers in the OR	Surgical exoskeleton	14	8	6	10-min	SUS
⁵⁴	Canada	Evaluation of robotic gloves related to improvement of hand function in daily tasks of life after stroke	Patients with hand disability	Robotic glove	9	-	-	Single session of 2 h	QUEST, and USE
¹	Canada	Evaluating the usability of a soft robotic glove for hand rehabilitation after stroke	Patients with hand disability	Soft robotic glove	14	10	4	60-to-90-min	SUS
⁵⁵	India	Designing and evaluating the usability of a hand rehabilitation	Patients with hand disability	Hand rehabilitation robot	45	-	-	Two-session training with the robot	SUS, and UEQ
⁵⁶	Spain	Evaluation of usability of MERLIN robotic system based on serious games for upper limb rehabilitation in stroke patients	people with stroke	Rehabilitation robot	9	6	3	3-sessions of 30 min	SUS, IMI, QUEST, and ArmAssist Usability Assessment Questionnaire
⁵⁷	Spain	Exploring the usability, acceptability, and adoption of a hybrid exoskeleton or robot designed for passive rehabilitation of the upper limbs	Patients with upper limp disabilities	Hybrid exoskeleton or robot for upper limb rehabilitation	7	1	6	-	SUS
⁵⁸	France	Development of a new gait rehabilitation device using RAGT and fully immersive VR evaluation of its usability	People with movement impairment and mild cognitive deficit	RAGT	18	5	13	20-min with 24 h separating the two sessions	SUS
⁵⁹	Colombia	Design and evaluation of lower limb rehabilitation robot	Patients with lower limb disabilities	Lower limb rehabilitation robot	5	2	3	-	SUS
⁶⁰	Switzerland	Evaluation the technical usability and feasibility of an interactive leg-press training robot	Children with neuromuscular impairments	Interactive leg press training robot	5	4	1	Between 40 and 45 min	SUS
⁶¹	Portugal	Usability evaluation of a upper-limb rehabilitation robot	Patient with upper-limb disabilities	Upper-limb rehabilitation robot	31	-	-	5-min	SUS and UEQ
⁶²	Germany	Investigating factors affecting user experience in robot-assisted health monitoring systems by combining subjective and objective health data	Patients’ caregivers	Robot-assisted health monitoring	8	-	-	Three sessions (30 and 60 min)	SUS and UEQ

SUS: System Usability Scale; PSSUQ: Post-Study System Usability Questionnaire; UEQ: User Experience Questionnaire; QUEST: Quebec User Evaluation of Satisfaction with Assistive Technology; MARS: Mobile Application Rating Scale; TDS: Tongue Drive System; TAM-FF: Technology Assessment Model Fast Form; FRAS: fall risk assessment system; IMI: Intrinsic Motivation Inventory; COPD: chronic obstructive pulmonary disease; ACT: acceptance and commitment therapy; GAD: generalized anxiety disorder; EMT: emergency medical technician; CLBP: chronic low back pain; IVR: immersive virtual reality; PD: Parkinson's disease; HMD: head-mounted display; TSQ-WT: Tele-healthcare Satisfaction Questionnaire for Wearable Technology; USEQ: User Satisfaction Evaluation Questionnaire; MCI: mild cognitive impairments; AD: Alzheimer Disease; OR: operating room; RAGT: robot-assisted gait training; VR: virtual reality; RAGT: A robot-assisted gait training.

Study distribution by year and country

As shown in Figure 2, the majority of the studies included in this study were conducted in Italy (n = 6 (wearables = 4, robots = 2), 12%) and United Kingdom (n = 5(wearables = 4, robots = 1), 10%) (More details in Table 2).

Figure 2.

Distribution of the studies based on country.

As shown in Figure 3, the majority of articles focusing on the most used questionnaires for evaluating the usability of robots and smart wearables were published in 2020 (n = 11),^{22–29,52–54} 2021 (n = 9),^{1,0–34,55–57} and 2022 (n = 10),^{35–41,58–60} respectively (More details in Table 2).

Figure 3.

Distribution of the studies in terms of publication year.

In the studies related to the evaluation of smart wearables, the largest sample size was 110 people, consisting of 60 women and 50 men.⁴⁰ Additionally, the smallest sample size in these studies included 4 people, comprising 2 women and 2 men.¹⁸ In evaluating the usability of robots, the largest sample size in the studies included 1860 people, with 919 women and 941 men.⁴⁸ In evaluating the usability of robots, the smallest sample size in these studies included 5 people, with the participation rate of men and women varying in these studies^45,59,60 (More details in Table 2).

The minimum and maximum intervention time for evaluating the usability of wearables was 5 minutes¹⁹ and 2 years,³⁰ respectively. In addition, the minimum and maximum intervention time for evaluating the usability of robots was 5 minutes⁶¹ and 6 months,⁵² respectively.

Evaluation questionnaire

The most frequently used questionnaires for evaluating the usability of smart wearables

The questionnaires used to evaluate the usability of wearables are shown in Table 3. Ten questionnaires were identified to evaluate the usability of wearables. System Usability Scale (SUS) (50%), Post-Study System Usability Questionnaire (PSSUQ) (19.44%), Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST) (8.33%), and Mobile Application Rating Scale (8.33%) were the most used questionnaires.

Table 3.

The most frequently used questionnaires for evaluating the usability of smart wearables.

Row	Questionnaire name	Ref	Frequency (%)	Description of the questionnaire
1	SUS	^{18, 20, 22–24, 26, 27, 31–38, 40–42}	18(50)	The SUS questionnaire was developed by Brooke et al. as a tool for evaluating the usability of electronic systems. It consists of ten items that are not categorized specifically. The items are rated on a 5-point Likert scale. The SUS questionnaire has been found to exhibit a high level of reliability, with a coefficient alpha of 0.91^{63, 64}.
2	PSSUQ	^{14–17, 19, 21, 25}	7(19.44)	The PSSUQ was developed by Lewis as a means to evaluate user satisfaction with system usability at the conclusion of a study. The latest version of the questionnaire comprises 16 questions categorized into three sections: system usefulness, information quality, and interface quality. These questions are rated on a 7-point Likert scale. The criterion validity of the PSSUQ has demonstrated a moderate correlation (r = 0.80) with other measures of user satisfaction. Additionally, the PSSUQ exhibits strong reliability, with a coefficient alpha of 0.96 ^{64, 65}.
3	QUEST	^{26, 30, 42}	3(8.33)	The QUEST is a comprehensive assessment tool that evaluates user satisfaction with assistive technology, focusing on two key components: Device and Services. Rigorous psychometric testing has been conducted to examine its test-retest stability, alternate-form equivalence, internal consistency, factorial composition, and nomological validity. It consists of various items that focus on measuring users’ opinions and experiences related to the effectiveness, ease of use, and overall satisfaction with the assistive technology they are utilizing ⁶⁶.
4	MARS	^{20, 28, 29}	3(8.33)	The MARS was introduced by Stoyanov et al. as a tool specifically designed to evaluate the quality of mobile health applications. This questionnaire consists of 23 items, categorized into six sections: engagement, functionality, aesthetics, information quality, subjective quality, and an app-specific section. The items in MARS are rated on a 5-point scale ⁶⁷. The construct validity of MARS was confirmed through confirmatory factor analysis, yielding favorable results (root mean square error of approximation = 0.074, Tucker–Lewis index = 0.922, confirmatory fit index = 0.940, standardized root mean square residual = 0.059). Furthermore, the reliability of the tool was verified with an Omega coefficient ranging from 0.79 to 0.93. In terms of concurrent validity, MARS demonstrated a significant correlation with ENLIGHT (p < 0.05), further supporting its validity as a measure for evaluating mobile health applications ⁶⁸.
5	IMI	²³	1(2.77)	The IMI is a questionnaire designed to assess an individual's intrinsic motivation, which refers to the internal drive and enjoyment derived from engaging in a particular activity. The IMI consists of several subscales that measure different aspects of intrinsic motivation, such as interest/enjoyment, perceived competence, effort/importance, value/usefulness, and pressure/tension. By capturing these dimensions, the IMI provides insights into the underlying factors that influence an individual's motivation to engage in a specific task or activity ⁶⁹.
6	TAM-FF	¹⁹	1(2.77)	The TAM-FF questionnaire (Davis, 1989) is a concise tool designed to evaluate users’ perceptions and attitudes towards technology. It assesses key dimensions such as perceived usefulness and perceived ease of use, which are crucial factors in determining individuals’ acceptance and adoption of technology. The TAM-FF questionnaire provides a quick and efficient means of gathering data on users’ perceptions, making it suitable for studies where time is limited ⁷⁰.
8	TSQ-WT	³⁹	1(2.77)	The TSQ-WT is a specialized questionnaire designed to assess user satisfaction with wearable technology used in tele-healthcare settings. It focuses on measuring user satisfaction with various aspects of wearable technology, including ease of use, functionality, reliability, and overall satisfaction with the tele-healthcare experience ³⁹. The TSQ-WT provides valuable insights into users’ perceptions and experiences with wearable technology, enabling researchers and practitioners to identify strengths, weaknesses, and areas for improvement in tele-healthcare interventions. TSQ-WT comprises six dimensions that assess different aspects of user satisfaction, including benefit, usability, self-concept, privacy and loss of control, QoL, and wearing comfort of the system. Each dimension consists of five items that user's rate on a 5-point Likert scale, ranging from 0 to 4 points. Higher scores on the scale indicate more positive ratings, reflecting a greater level of satisfaction with the specific dimension being measured in the questionnaire ⁷¹.
9	NASA TLX	²²	1(2.77)	The NASA TLX questionnaire is a widely used tool for assessing subjective workload and mental demands experienced by individuals during a task. It measures workload across six dimensions, including mental demand, physical demand, temporal demand, performance, effort, and frustration. Participants rate each dimension on a scale from 0 to 100, indicating the perceived level of workload or demand. The TLX questionnaire provides valuable insights into the cognitive and physical demands placed on individuals during various tasks, allowing researchers and practitioners to evaluate and compare workload across different conditions or interventions ⁷².
10	USEQ	⁴⁰	1(2.77)	The USEQ is a questionnaire designed to assess user satisfaction with a specific product, service, or system. It aims to capture users’ perceptions, opinions, and overall satisfaction regarding their experience. The USEQ typically consists of multiple items or statements that users rate on a scale, reflecting their level of satisfaction. The questionnaire covers various dimensions such as usability, functionality, aesthetics, and overall satisfaction. The USEQ is a reliable tool for measuring user satisfaction, as its psychometric properties have been thoroughly tested and validated. Its reliability ensures consistent and accurate measurement of user satisfaction, allowing researchers and practitioners to gather valuable insights for improving the design, usability, and overall quality of the product or service under evaluation. The USEQ, consisting of six items, demonstrated significant associations among them, indicating internal consistency. The Cronbach's alpha coefficient for the questionnaire was calculated to be 0.716, indicating a moderate level of reliability ⁷³.

SUS: System Usability Scale; PSSUQ: Post-Study System Usability Questionnaire; MARS: Mobile Application Rating Scale; IMI: Intrinsic Motivation Inventory; TAM-FF: Technology Assessment Model Fast Form; TSQ-WT: Tele-healthcare Satisfaction Questionnaire for Wearable Technology; QoL: quality of life; TLX: Task Load Index; USEQ: User Satisfaction Evaluation Questionnaire.

The most frequently used questionnaires for evaluating the usability of robots

Seven questionnaires were identified for evaluating the usability of robots (Table 4). SUS (56.66%), User Experience Questionnaire (UEQ) (16.66%), and QUEST (10%) were the most used questionnaires for evaluating the usability of robots.

Table 4.

The most frequently used questionnaires for evaluating the usability of robots.

Row	Questionnaire name	Ref	Frequency (%)	Description of the questionnaire
1	SUS	^{1, 43–46, 49, 51–53, 55–62}	17(56.66)	Table 3 provides a comprehensive overview of the questionnaire's description.
2	UEQ	^{50, 52, 55, 61, 62}	5(16.66)	The UEQ was introduced by Laugwitz et al.⁷⁴, as a tool for evaluating usability and user satisfaction. The questionnaire consists of items categorized into six sections: attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty. The items in this questionnaire are rated using a 7-point scale. The factor analysis of the UEQ demonstrated satisfactory validity, indicating that the questionnaire measures the intended constructs effectively. Additionally, the reliability of the UEQ was confirmed through Cronbach's alpha analysis, which exceeded 0.71, indicating strong internal consistency ⁷⁴.
3	QUEST	^{48, 54, 56}	3(10)	Table 3 provides a comprehensive overview of the questionnaire's description.
4	USE	^{47, 54}	2(6.66)	The USE questionnaire was developed by Lund et al. as a tool for assessing the usability of a system's user interface. This questionnaire consists of 30 items divided into four sections: usefulness, ease of use, ease of learning, and satisfaction. The items are rated on a 7-point Likert scale ⁷⁵. The validity of the USE questionnaire was demonstrated by the high correlations between its dimensions and the SUS, with correlation coefficients ranging from 0.60 to 0.82 (p < 0.001). These findings indicate that the USE questionnaire effectively measures usability. Additionally, the reliability of the USE questionnaire has been confirmed with a high Cronbach's alpha coefficient of 0.98, indicating strong internal consistency⁷⁵.
5	IMI	⁵⁶	1(3.33)	Table 3 provides a comprehensive overview of the questionnaire's description.
6	TSQ-WT	⁴³	1(3.33)	Table 3 provides a comprehensive overview of the questionnaire's description.
7	ArmAssist Usability Assessment Questionnaire	⁵⁶	1(3.33)	The ArmAssist Usability Assessment Questionnaire is a specific questionnaire designed to evaluate the usability of the ArmAssist device. It assesses various usability dimensions, including ease of use, learnability, efficiency, effectiveness, and user satisfaction. The questionnaire comprises 17 survey items and is specifically tailored for use with the AA device, which is utilized in the MERLIN system. Participants, including patients and therapists, rate the questions on a scale of 1 (strongly agree) to 7 (strongly disagree), providing feedback on their satisfaction with the system and therapy. Additionally, the questionnaire includes three open-ended questions that allow participants to share their subjective opinions, such as the aspects they liked the most, identified negative aspects, and proposals for system improvement⁵⁶.

SUS: System Usability Scale; UEQ: User Experience Questionnaire; QUEST: Quebec User Evaluation of Satisfaction with Assistive Technology; USE: Usefulness, Satisfaction, and Ease of Use Questionnaire; IMI: Intrinsic Motivation Inventory; TSQ-WT: Tele-healthcare Satisfaction Questionnaire for Wearable Technology.

Quality appraisal

The results of the studies’ quality evaluation are outlined in Appendix B.

Discussion

This paper presents an identification of the most prevalent and extensively employed questionnaires utilized for assessing the usability of robots and smart wearables. The SUS and PSSUQ have emerged as the primary questionnaires used to assess the usability of smart wearables. Additionally, SUS, UEQ, and QUEST have been identified as the prevailing questionnaires employed to evaluate the usability of robots.

In the review conducted by Hajesmaeel-Gohari et al.,⁹ the SUS and PSSUQ emerged as two commonly used questionnaires to assess satisfaction, usability, acceptance, and quality outcomes in the field of mobile health. According to their beliefs, the SUS is a widely used questionnaire for assessing the usability of electronic systems. In comparison to other questionnaires like CSUQ, SUS is considered a faster tool for evaluating perceived usability, as it contains fewer items and a simpler scale. Additionally, the SUS questionnaire includes a satisfaction question, which is often evaluated separately in dedicated satisfaction assessment tools but is encompassed within the usability evaluation. Due to its features, reproducibility, reliability, and validity, researchers and evaluators of mHealth services frequently employ the SUS questionnaire,⁹ Kaya et al.,⁷⁶ also asserted that the SUS incorporating an adjective scale rating stands out as a widely embraced and user-friendly questionnaire for assessing the usability of various products. This approach offers a straightforward score calculation, providing a comprehensive snapshot of a product's usability. Other studies also demonstrate that, with its straightforward scoring system, SUS facilitates efficient and rapid administration, rendering it a cost-effective option for usability testing.⁷⁷ The questionnaire's enduring presence in usability evaluation enables longitudinal comparisons, and its established credibility within the field enhances the trustworthiness of the insights it generates.⁷⁸ The assertion by Kaya et al.,⁷⁶ further solidifies SUS as a user-friendly and versatile tool for assessing various product usability. Additionally, the findings of other studies affirm SUS's efficiency in terms of scoring, administration, and cost-effectiveness, while its enduring presence allows for meaningful longitudinal comparisons. Overall, the extensive support from multiple studies underscores the credibility and utility of the SUS questionnaire in providing valuable insights for usability evaluations and beyond. It is recommended that future research explores potential enhancements or adaptations to further optimize its applicability in evolving technological landscapes.

Moreover, PSSUQ provides a comprehensive assessment of system usability, capturing users’ perceptions of different aspects such as efficiency, learnability, and satisfaction, allowing for a holistic evaluation. The questionnaire is a well-established and validated tool, ensuring reliable and valid results, providing accurate insights into the usability of the system.⁹ The study by Vlachogianni and Tselios,⁷⁹ shows that the comprehensive assessment approach of PSSUQ, covering efficiency, learnability, and satisfaction, ensures a thorough understanding of the user experience with these devices. Moreover, the PSSUQ is designed to be a practical and efficient tool, requiring users to respond to a limited number of items, making it feasible for use in various research and evaluation settings, enabling timely and valuable usability feedback.^9,64,65 Researchers and designers can rely on these questionnaires to obtain reliable and comparable usability data for smart wearables, enabling them to make informed decisions regarding improvements or optimizations. Moreover, in different studies,⁸⁰ the utility of PSSUQ is particularly emphasized in the context of smart wearables, where researchers and designers can depend on its reliability to obtain comparable usability data. Additionally, researchers and smart wearable device designers can depend on PSSUQ to obtain reliable and comparable usability data, enabling informed decision-making for improvements or optimizations. Moreover, PSSUQ's adaptability allows for exploration of potential refinements, ensuring its continued applicability and relevance in the ever-evolving landscape of smart wearables. As future developments in smart wearable technology unfold, it is recommended that researchers explore potential refinements or adaptations to ensure the continued applicability and relevance of PSSUQ in evolving technological landscapes.

Additionally, as emphasized in the study, the importance of the SUS, UEQ, and QUEST becomes evident when assessing the usability of robots. These questionnaires have emerged as the prevailing choices among researchers and practitioners, emphasizing their suitability and reliability in assessing the usability aspects specific to robotics.^8,9 SUS simplicity and efficiency make it a widely adopted tool, particularly valuable for assessing user experiences with robotic systems. The standardized nature of SUS enables consistent and comparable evaluations across different robots, fostering reliable insights into their usability.^46,59,60,62 Hesen⁸¹ believed that, with a concise set of items, SUS minimizes respondent burden, ensuring quick and straightforward assessments. The questionnaire's versatility allows it to be applied to various robotic interfaces and applications, making it adaptable to the diverse functionalities of robotic systems.⁸² Additionally, the SUS's scoring system provides a quantitative measure of usability, aiding in the comparison of different robots. As a well-established and validated tool, SUS has gained credibility in the field, offering a reliable means for designers and researchers to gather valuable feedback, identify usability issues, and make informed decisions to optimize the user experience with robotic technologies.

The UEQ, with its simplicity and comprehensiveness, provides a holistic assessment of user experience by measuring key dimensions like attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty.⁹ It offers a standardized approach that allows for reliable comparisons across different interfaces or products. Fitriana et al.,⁸³ believed that the comprehensive nature of UEQ allows for the assessment of diverse aspects such as attractiveness, efficiency, novelty, and dependability, providing a holistic perspective on the user's interaction with a product or system. Hussain et al.,⁸⁴ also mention that the standardized format of UEQ facilitates consistent evaluations across various contexts, contributing to reliable and comparable results. With its user-friendly design, UEQ ensures accessibility for a wide range of respondents, thereby promoting high response rates and yielding valuable insights. Additionally, the questionnaire's flexibility allows it to be adapted to different interfaces and applications, enhancing its applicability across diverse domains. UEQ's ability to generate actionable feedback makes it a valuable tool for designers and researchers, enabling informed decision-making to optimize and enhance user experiences. Moreover, the flexibility of the questionnaire enables adaptation to various interfaces and applications, thereby enhancing its relevance across diverse domains for robotic systems. The UEQ's capacity to generate actionable feedback establishes it as a valuable tool for robotic designers and researchers, facilitating informed decision-making to optimize and enhance user interactions with robotic technologies.⁸⁵

On the other hand, QUEST focuses on specific aspects such as user satisfaction, efficiency, and engagement, providing valuable insights into the usability and user interaction of a system.⁸⁶ These questionnaires offer a standardized and reliable approach for assessing the usability aspects specific to health systems, making them essential tools in this field of research. This evaluation tool is specifically tailored for assistive technology, making it uniquely suited for appraising robots designed to aid individuals. QUEST's structured format and focus on user satisfaction allow for a comprehensive understanding of how well a robot fulfills its intended purpose. Its well-defined categories, such as effectiveness, ease of use, and satisfaction, facilitate a detailed analysis of user experiences.^48,54,56 Demers et al.,⁸⁷ believed that QUEST offers a standardized and validated framework, ensuring reliability and consistency in evaluations. The adaptability of QUEST to different assistive robotic systems contributes to its versatility in capturing varied user perspectives.^48,56 With its emphasis on user satisfaction and its established reliability, QUEST stands out as a valuable instrument for gaining insights into the performance and impact of assistive robots, aiding designers and researchers in refining and optimizing these technologies to better meet user needs. In conclusion, the adaptability of QUEST to diverse assistive robotic systems further enhances its versatility, capturing varied user perspectives. With a strong emphasis on user satisfaction and a proven track record of reliability, QUEST emerges as an indispensable tool for discerning the performance and impact of assistive robots. It serves as a guiding resource for designers and researchers, aiding in the continual refinement and optimization of these technologies to better address user needs in the ever-evolving landscape of health systems and assistive robotics.

In the end, it should be mentioned that by understanding the questionnaires commonly utilized in these domains, stakeholders involved in the development and evaluation of robots and smart wearables can benefit from a standardized approach. Consistency in usability assessment enables benchmarking, comparison of results, and identification of best practices. Moreover, the utilization of well-established questionnaires facilitates knowledge sharing and collaboration among researchers, leading to the advancement of usability evaluation methodologies.

Limitations of the study

There were a few limitations in our study. Firstly, the inclusion criteria for this study only considered articles published in English, potentially excluding relevant studies published in other languages. Additionally, the search for related studies was conducted using three scientific databases: Scopus, PubMed, and Web of Science. To ensure more comprehensive results, it is recommended that future studies include articles published in other languages and a broader range of databases. Furthermore, it is important to note that this study did not conduct a critical appraisal of individual sources of evidence, and this limitation should be addressed in future studies.

Conclusion

This paper has identified the most common and widely used questionnaires employed in the evaluation of the usability of robots and smart wearables. The SUS and Post-Study System Usability Questionnaire (PSSUQ) emerged as the predominant questionnaires utilized for assessing the usability of smart wearables. Furthermore, the SUS, UEQ, and Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST) were identified as the most commonly used questionnaires for evaluating the usability of robots.

The utilization of these questionnaires plays a crucial role in enhancing the design and user experience of robots and smart wearables. By incorporating these widely used evaluation tools, researchers and practitioners can gain valuable insights into the strengths, weaknesses, and areas for improvement. This knowledge can drive advancements in creating more user-friendly and efficient technologies within the dynamic realm of robotics and smart wearables. Continued exploration and application of these questionnaires will contribute to the ongoing evolution of these technologies, ultimately benefiting users and enhancing their overall satisfaction.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241237384 - Supplemental material for The most used questionnaires for evaluating the usability of robots and smart wearables: A scoping review

Supplemental material, sj-docx-1-dhj-10.1177_20552076241237384 for The most used questionnaires for evaluating the usability of robots and smart wearables: A scoping review by Khadijeh Moulaei, Reza Moulaei and Kambiz Bahaadinbeigy in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076241237384 - Supplemental material for The most used questionnaires for evaluating the usability of robots and smart wearables: A scoping review

Supplemental material, sj-docx-2-dhj-10.1177_20552076241237384 for The most used questionnaires for evaluating the usability of robots and smart wearables: A scoping review by Khadijeh Moulaei, Reza Moulaei and Kambiz Bahaadinbeigy in DIGITAL HEALTH

Footnotes

Acknowledgments

The authors thank the Central Library and Documentation Center of Kerman University of Medical Sciences for providing access to knowledge base references required for this study.

Authors’ contributions

Khadijeh Moulaei contributed to the conceptualization;data curation;formal analysis;investigation;methodology;project administration;writing—original draft;and writing—review and editing. Reza Moulaei contributed to the conceptualization;data curation;formal analysis;investigation;methodology;resources;Kambiz Bahaadinbeigy contributed to the investigation;writing—review and editing.

Availability of data and material

All data relevant to the study are included in the article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The authors received no financial support for the research,authorship,and/or publication of this article.

Guarantor

KHM,KB.

Supplemental material

Supplemental material for this article is available online.

References

Proulx

Higgins

Gagnon

. Occupational therapists’ evaluation of the perceived usability and utility of wearable soft robotic exoskeleton gloves for hand function rehabilitation following a stroke. Disabil Rehabil: Assist Technol 2023; 18: 953–962.

Meyer

Tanczak

Kanzler

, et al. Design and validation of a novel online platform to support the usability evaluation of wearable robotic devices. Wearable Technol 2023; 4: e3. Jia Y, Wang W, Wen D, et al. Perceived user preferences and usability evaluation of mainstream wearable devices for health monitoring. PeerJ 2018; 6: e5350.

Scott

. The practice of usability: teaching user engagement through service-learning. Tech Commun Q 2008; 17: 381–412.

Nielsen

. Usability engineering. San Francisco, CA, USA: Morgan Kaufmann, 1994.

Cempini

Marzegan

Rabuffetti

, et al. Analysis of relative displacement between the HX wearable robotic exoskeleton and the user’s hand. J Neuroeng Rehabil 2014; 11: 1–8.

Meyer

Gassert

Lambercy

. An analysis of usability evaluation practices and contexts of use in wearable robotics. J Neuroeng Rehabil 2021; 18: 1–15.

Khakurel

Porras

Melkas

, et al. A comprehensive framework of usability issues related to the wearable devices. In: Convergence of ICT and smart devices for emerging applications. Cham: Springer, 2020, pp.21–66.

Hajesmaeel-Gohari

Bahaadinbeigy

. The most used questionnaires for evaluating telemedicine services. BMC Med Inform Decis Mak 2021; 21: 1–11.

Hajesmaeel-Gohari

Khordastan

Fatehi

, et al. The most used questionnaires for evaluating satisfaction, usability, acceptance, and quality outcomes of mobile health. BMC Med Inform Decis Mak 2022; 22: 22.

10.

Hyzy

Bond

Mulvenna

, et al. System usability scale benchmarking for digital health apps: meta-analysis. JMIR mHealth uHealth 2022; 10: e37290.

11.

Papanastasiou

Kousi

Karagiannis

, et al. Towards seamless human robot collaboration: integrating multimodal interaction. Int J Adv Manuf Technol 2019; 105: 3881–3897.

12.

Hong

Fàbregues

Bartlett

, et al. The mixed methods appraisal tool (MMAT) version 2018 for information professionals and researchers. Educ Inf 2018; 34: 285–291.

13.

Hong

Pluye

Fàbregues

, et al. Mixed methods appraisal tool (MMAT), version 2018. Registration of Copyright, 1148552. 2018, p.10.

14.

Dosani

Hunc

Dumont

, et al. A vibro-tactile display for clinical monitoring: real-time evaluation. Anesth Analg 2012; 115: 588–594.

15.

Kim

Park

Bruce

, et al. Qualitative assessment of Tongue Drive System by people with high-level spinal cord injury. J Rehabil Res Dev 2014; 51: 451–466.

16.

Ferreira

Godinho

Santos

, et al. Quantitative home-based assessment of Parkinson's symptoms: the SENSE-PARK feasibility and usability study. BMC Neurol 2015; 15. DOI: https://doi.org/10.1186/s12883-015-0343-z

17.

Meritam

Ryvlin

Beniczky

. User-based evaluation of applicability and usability of a wearable accelerometer device for detecting bilateral tonic-clonic seizures: a field study. Epilepsia 2018; 59: 48–52.

18.

Botros

Schutz

Camenzind

, et al. Long-term home-monitoring sensor technology in patients with Parkinson's disease-acceptance and adherence. Sensors 2019; 19: 5169.

19.

Matcham

Barattieri di San Pietro

Bulgari

, et al. Remote assessment of disease and relapse in major depressive disorder (RADAR-MDD): a multi-centre prospective cohort study protocol. BMC Psychiatry 2019; 19: 72.

20.

Rehman

RZU

Jang

Xiong

. Usability evaluations of a newly developed wearable inertial sensing system for assessing elderly fall risk. Adv Intell Syst Comput 2019; 789: 423–434.

21.

Albani

Ferraris

Nerino

, et al. An integrated multi-sensor approach for the remote monitoring of Parkinson's disease. Sensors 2019; 19: 4764.

22.

Lambelet

Temiraliuly

Siegenthaler

, et al. Characterization and wearability evaluation of a fully portable wrist exoskeleton for unsupervised training after stroke. J Neuroeng Rehabil 2020; 17. DOI: https://doi.org/10.1186/s12984-020-00749-4.

23.

Keogh

Dorn

Walsh

, et al. Comparing the usability and acceptability of wearable sensors among older Irish adults in a real-world context: observational study. JMIR mHealth and uHealth 2020; 8(4): e15704.

24.

Weenk

Bredie

Koeneman

, et al. Continuous monitoring of vital signs in the general ward using wearable devices: randomized controlled trial. J Med Internet Res 2020; 22: e15471.

25.

Bruno

Biondi

Bottcher

, et al. Remote assessment of disease and relapse in epilepsy: protocol for a multicenter prospective cohort study. JMIR Res Protoc 2020; 9: e21840.

26.

van Dijsseldonk

van Nes

IJW

Geurts

ACH

, et al. Exoskeleton home and community use in people with complete spinal cord injury. Sci Rep 2020; 10. DOI: https://doi.org/10.1038/s41598-020-72397-6.

27.

Bentley

Powell

Potter

, et al. The use of a smartphone app and an activity tracker to promote physical activity in the management of chronic obstructive pulmonary disease: randomized controlled feasibility study. JMIR mHealth uHealth 2020; 8(6): e16203.

28.

Hafiz

Bardram

. The ubiquitous cognitive assessment tool for smartwatches: design, implementation, and evaluation study. JMIR mHealth uHealth 2020; 8: e17506.

29.

Hafiz

Maxhuni

Bardram

. Analysis of perceived human factors and participants’ demographics during a cognitive assessment study with a smartwatch. In: 2020 IEEE International Conference On Healthcare Informatics (ICHI), 2020. DOI: https://doi.org/10.1109/ICHI48887.2020.9374342.

30.

Piau

Steinmeyer

Charlon

, et al. A smart shoe insole to monitor frail older adults’ walking speed: results of two evaluation phases completed in a living lab and through a 12-week pilot study. JMIR mHealth uHealth 2021; 9: e15641.

31.

Hemmings

Kawadler

Whatmough

, et al. Development and feasibility of a digital acceptance and commitment therapy-based intervention for generalized anxiety disorder: pilot acceptability study. JMIR Form Res 2021; 5: e21737.

32.

ter Harmsel

van der Pol

Swinkels

, et al. Development of a wearable biocueing app (Sense-IT) among forensic psychiatric outpatients with aggressive behavior: design and evaluation study. JMIR Form Res 2021; 5: e29267.

33.

Longatelli

Antonietti

Biffi

, et al. User-centred assistive SystEm for arm functions in neUromuscuLar subjects (USEFUL): a randomized controlled study. J Neuroeng Rehabil 2021; 18: 4.

34.

Yoon

Kim

Roh

, et al. Twelve-lead electrocardiogram acquisition with a patchy-type wireless device in ambulance transport: simulation-based randomized controlled trial. JMIR mHealth uHealth 2021; 9: e24142.

35.

Kim

Choi

. Mobile health platform based on user-centered design to promote exercise for patients with peripheral artery disease. BMC Med Inform Decis Mak 2022; 22. DOI: https://doi.org/10.1186/s12911-022-01945-z.

36.

Blanchard

Nguyen

Devanne

, et al. Technical feasibility of supervision of stretching exercises by a humanoid robot coach for chronic low back pain: the R-COOL randomized trial. BioMed Res Int 2022; 2022: 5667223.

37.

Semprini

Lencioni

Hinterlang

, et al. User-centered design and development of TWIN-Acta: a novel control suite of the TWIN lower limb exoskeleton for the rehabilitation of persons post-stroke. Front Neurosci 2022; 16. DOI: https://doi.org/10.3389/fnins.2022.915707.

38.

Campo-Prieto

Cancela-Carral

Rodríguez-Fuentes

. Wearable immersive virtual reality device for promoting physical activity in Parkinson's disease patients. Sensors (Basel, Switzerland) 2022; 22: 3302.

39.

Schmidt

Jansen

Depenbusch

, et al. Using wearables to promote physical activity in old age: feasibility, benefits, and user friendliness. Z Gerontol Geriatr 2022; 55: 388–393.

40.

Domingos

Costa

Santos

, et al. Usability, acceptability, and satisfaction of a wearable activity tracker in older adults: observational study in a real-life context in Northern Portugal. J Med Internet Res 2022; 24: e26652.

41.

Virbel-Fleischman

Rétory

Hardy

, et al. Body-worn sensors for Parkinson's disease: a qualitative approach with patients and healthcare professionals. PLoS One 2022; 17: e0265438.

42.

Amore

Silvestri

Guidobaldi

, et al. Efficacy and patients’ satisfaction with the ORCAM MyEye device among visually impaired people: a multicenter study. J Med Syst 2023; 47. DOI: https://doi.org/10.1007/s10916-023-01908-5.

43.

Ambrosini

Ferrante

Rossini

, et al. Functional and usability assessment of a robotic exoskeleton arm to support activities of daily life. Robotica 2014; 32(8): 1213–1224.

44.

Syrdal

Dautenhahn

Koay

, et al. Views from within a narrative: evaluating long-term human-robot interaction in a naturalistic environment using open-ended scenarios. Cognit Comput 2014; 6: 741–759.

45.

Vanmulken

Spooren

Bongers

, et al. Robot-assisted task-oriented upper extremity skill training in cervical spinal cord injury: a feasibility study. Spinal Cord 2015; 53: 547–551.

46.

Lopez-Samaniego

Garcia-Zapirain

. A robot-based tool for physical and cognitive rehabilitation of elderly people using biofeedback. Int J Environ Res Public Health 2016; 13: 1176.

47.

Swinnen

Lefeber

Willaert

, et al. Motivation, expectations, and usability of a driven gait orthosis in stroke patients and their therapists. Top Stroke Rehabil 2017; 24: 299–308.

48.

Schliessmann

Nisser

Schuld

, et al. Trainer in a pocket - proof-of-concept of mobile, real-time, foot kinematics feedback for gait pattern normalization in individuals after stroke, incomplete spinal cord injury and elderly patients. J Neuroeng Rehabil 2018; 15. DOI: https://doi.org/10.1186/s12984-018-0389-4.

49.

Valdés

Van der Loos

HFM

. Biofeedback vs. game scores for reducing trunk compensation after stroke: a randomized crossover trial. Top Stroke Rehabil 2018; 25: 96–113.

50.

Gerlowska

Skrobas

Grabowska-Aleksandrowicz

, et al. Assessment of perceived attractiveness, usability, and societal impact of a multimodal robotic assistant for aging patients with memory impairments. Front Neurol 2018; 9. DOI: https://doi.org/10.3389/fneur.2018.00392.

51.

Eicher

Haesner

Spranger

, et al. Usability and acceptability by a younger and older user group regarding a mobile robot-supported gait rehabilitation system. Assist Technol: Off J RESNA 2019; 31: 25–33.

52.

Pérez-Rodríguez

Moreno-Sánchez

Valdés-Aragonés

, et al. Friwalk robotic walker: usability, acceptance and UX evaluation after a pilot study in a real environment. Disabil Rehabil Assist Technol 2020; 15: 718–727.

53.

Cha

Monfared

Stefanidis

, et al. Supporting surgical teams: identifying needs and barriers for exoskeleton implementation in the operating room. Hum Factors 2020; 62: 377–390.

54.

Yurkewich

Kozak

Ivanovic

, et al. Myoelectric untethered robotic glove enhances hand function and performance on daily living tasks after stroke. J Rehabil Assist Technol Eng 2020; 7: 205566832096405.

55.

Nehrujee

Andrew

Reethajanetsurekha , et al. Plug-and-train robot (PLUTO) for hand rehabilitation: design and preliminary evaluation. IEEE Access 2021; 9: 134957–134971.

56.

Guillen-Climent

Garzo

Munoz-Alcaraz

, et al. A usability study in patients with stroke using MERLIN, a robotic system based on serious games for upper limb rehabilitation in the home setting. J Neuroeng Rehabil 2021; 18. DOI: https://doi.org/10.1186/s12984-021-00837-z.

57.

Alguacil-Diego

Cuesta-Gomez

Contreras-Gonzalez

, et al. Validation of a hybrid exoskeleton for upper limb rehabilitation. A preliminary study. Sensors 2021; 21(21): 7342.

58.

Morizio

Compagnat

Boujut

, et al. Immersive virtual reality during robot-assisted gait training: validation of a new device in stroke rehabilitation. Medicina (Kaunas, Lithuania) 2022; 58: 1805.

59.

Pérez

Yepes

Vargas

, et al. Virtual reality game for physical and emotional rehabilitation of landmine victims. Sensors (Basel, Switzerland) 2022; 22: 5602.

60.

Chrif

van Hedel

HJA

Vivian

, et al. Usability evaluation of an interactive leg press training robot for children with neuromuscular impairments. Technol Health Care 2022; 30: 1183–1197.

61.

Rodrigues

Menezes

Restivo

. An augmented reality interface to control a collaborative robot in rehab: a preliminary usability evaluation. Front Digit Health 2023; 5. DOI: https://doi.org/10.3389/fdgth.2023.1078511.

62.

Neef

Linden

Richert

. Exploring the influencing factors on user experience in robot-assisted health monitoring systems combining subjective and objective health data. Appl Sci-Basel 2023; 13: 3537.

63.

Lewis

. The system usability scale: past, present, and future. Int J Hum–Comput Interact 2018; 34: 577–590.

64.

Brooke

. Sus: a quick and dirty usability. Usability Eval Ind 1996; 189: 189–194.

65.

Lewis

. IBM Computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum–Comput Interact 1995; 7: 57–78.

66.

Demers

Weiss-Lambrou

Ska

. Development of the Quebec user evaluation of satisfaction with assistive technology (QUEST). Assist Technol 1996; 8: 3–13.

67.

Stoyanov

Hides

Kavanagh

, et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth 2015; 3: e3422.

68.

Terhorst

Philippi

Sander

, et al. Validation of the mobile application rating scale (MARS). PLoS One 2020; 15: e0241480.

69.

Inventory

. Intrinsic motivation inventory (IMI). The Intrinsic Motivation Inventory, Scale description. 1994, pp.1–3.

70.

Davis

. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989; 13: 319–340.

71.

Werner

Geravand

Korondi

, et al. Evaluating the sit-to-stand transfer assistance from a smart walker in older adults with motor impairments. Geriatr Gerontol Int 2020; 20: 312–316.

72.

Hart

. NASA-task load index (NASA-TLX); 20 years later. Proc Hum Factors Ergon Soc Annu Meet 2006; 50: 904–908.

73.

Gil-Gómez

Manzano-Hernández

Albiol-Pérez

, et al. USEQ: a short questionnaire for satisfaction evaluation of virtual rehabilitation systems. Sensors (Basel, Switzerland) 2017; 17: 1589.

74.

Laugwitz

Held

Schrepp

. Construction and evaluation of a user experience questionnaire. In: HCI and usability for education and work: 4th symposium of the workgroup human-computer interaction and usability engineering of the austrian computer society, USAB 2008, Graz, Austria, 20–21 November 2008, pp.63–76. Springer.

75.

Gao

Kortum

Oswald

. Psychometric evaluation of the use (usefulness, satisfaction, and ease of use) questionnaire for reliability and validity. Proc Hum Factors Ergon Soc Annu Meet 2018; 62: 1414–1418.

76.

Kaya

Ozturk

Altin Gumussoy

. Usability measurement of mobile applications with system usability scale (SUS). In: Industrial engineering in the big data era: selected papers from the global joint conference on industrial engineering and its application areas, GJCIE 2018, Nevsehir, Turkey, 21–22 June 2018, pp.389–400. Springer, 2019.

77.

Ratnawati

Widianingsih

Anggraini

, et al. Evaluation of digital library's usability using the system usability scale method of (a case study). In: 2020 8th International Conference on Cyber And It Service Management (CITSM), 2020, pp.1–5: IEEE.

78.

McLellan

Muddimer

Peres

. The effect of experience on system usability scale ratings. J Usability Stud 2012; 7: 56–67.

79.

Vlachogianni

Tselios

. Perceived usability evaluation of educational technology using the post-study system usability questionnaire (PSSUQ): a systematic review. Sustainability 2023; 15: 12954.

80.

Bhanvadia

Brar

Delavar

, et al. Assessing usability of smartwatch digital health devices for home blood pressure monitoring among glaucoma patients. Informatics 2022; 9: 79.

81.

Hesen

. Investigating and improving the system usability scale for usability evaluation in practice.

82.

Martini

Broadbent

Koo

, et al. Investigating the usability, efficacy and accuracy of a medication entering software system for a healthcare robot. Front Robot AI 2022; 9: 814268.

83.

Fitriana

Iftadi

Astuti

. User experience evaluation and interface redesign using user experience questionnaire and Google material design (case study: SIAKAD UNS). Jurnal Teknik Industri 2023; 13: 159–164.

84.

Hussain

Ul Hassan

Muhammad Bilal

, et al. Model-based adaptive user interface based on context and user experience evaluation. J Multimodal User Interfaces 2018; 12: 1–16.

85.

Apraiz

Lasa

Mazmela

. Evaluation of user experience in human–robot interaction: a systematic literature review. Int J Soc Robot 2023; 15: 187–210.

86.

Demers

Weiss-Lambrou

Ska

. Quebec user evaluation of satisfaction with assistive technology versione 2.0. The Institute for Matching Persons and Technology: Webster, NY, 2000.

87.

Demers

Weiss-Lambrou

Ska

. The Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 2.0): an overview and recent progress. Technol Disabil 2002; 14: 101–105.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.03 MB

0.00 MB

0.10 MB