Sage Journals: Discover world-class research

Abstract

Background

Systematically developed quality indicators (QIs) facilitate the measurement and monitoring of quality of care and ultimately meaningful quality improvement. The aim of this study was to evaluate the acceptability of a predetermined suite of 84 scientifically valid prehospital care QIs from the provider perspective.

Methods

An explanatory sequential mixed methods study design was used. Quantitative data were obtained from 36 participants of an online survey in which they rated the acceptability of the QIs using a five-point numerical rating scale. Qualitative data were gathered by conducting semi-structured interviews with a purposive sample of nine survey participants. The successional collection of quantitative and qualitative data facilitated integrated interpretations and conclusions about the acceptability of the QIs.

Results

Generally, the acceptability of all QIs in the suite was rated highly. Data suggested a positive association between acceptability and other key characteristics of QIs. QIs that were seen as clear, supported by scientific evidence, practical and meaningful tended to be more acceptable than those which were not. The benefits of outcome type QIs were recognised, but participants raised concerns about their sensitivity in the measurement of prehospital quality of care. To be acceptable, QIs which included time intervals needed to be specific about time-critical interventions. Further, the high acceptability of the QIs was explained by a connection to participants’ professional values and qualities. Assessing the QIs’ acceptability from a healthcare provider perspective meant that QIs on patient satisfaction frequently received lower ratings.

Conclusion

The findings of this study provide evidence of the acceptability of prehospital care providers of a proposed suite of QIs. Future research should evaluate the feasibility and reliability of the QIs. There is also a need to investigate how acceptable the proposed QIs are to patients and communities.

Keywords

quality indicators quality improvement prehospital care

Background

The measurement of performance is integral to quality management within and across healthcare organisations and systems. Measuring and monitoring the quality of care and services starts with the development of quality indicators (QIs) of desirable performance and outcomes.¹ Quality improvement has experienced extensive growth leading to the emergence of improvement as a science in itself.^2–4 This has advanced the scientific rigour around the approaches and methods used for selecting and developing single QIs addressing specific aspects of quality, and QI suites addressing broader areas such as prehospital care.^5–7 Whilst scientific validity is a minimum prerequisite for any QI, subsequent developmental work should aim to provide empirical evidence, as far as possible, of a number of other key characteristics, including acceptability.^5,6,8,9 Acceptability is a multi-faceted construct,¹⁰ and in the development and application of QIs depends on the extent to which measurement of performance based on a particular QI is acceptable to both those being assessed and those undertaking the assessment.⁸ Quality measurement is not synonymous with quality improvement. However, positive change cannot occur without meaningful measurement of performance.² For measurement to be effective in facilitating improvement the gathered intelligence needs to be able to influence decision-makers. If decision-makers and key stakeholders do not accept a QI, the results of associated measurement will not be useful for influencing people to make a change.⁵ Therefore, the potential of a QI to facilitate quality improvement relies on it being acceptable to stakeholders.

An all-inclusive definition of prehospital care comprises all healthcare services and resources prior to referral to a hospital, if needed. However, for the purpose of this project, prehospital care was confined to the care and services provided by ambulance services. In Australia, the provision of prehospital care is performed predominantly by jurisdictional state/territory ambulance services forming an important part of the national health system. In 2020/2021, these ambulance services had an operational workforce consisting of 16,742 staff and responded to over 4 million incidents (158.7 per 1,000 population).¹¹ Ambulance services in Australia, like most other healthcare services anywhere, are under pressure to maintain contemporary, high-quality patient care in an environment with constantly growing demands and complexity.^12,13 The right measurement of the right data over time, and its use as performance intelligence, plays a pivotal role in guiding any healthcare services’ decision-makers with respect to quality of care.¹⁴

Whilst a plethora of healthcare QIs exist,¹⁵ relatively few have been developed specifically for prehospital care.¹⁶ Moreover, of these prehospital care QIs many have a focus on operational aspects (such as response times) and have been developed using non-systematic methods.¹⁶ Notwithstanding more sophisticated service-specific QIs developed by Australian state/territory ambulance services, the quality of care delivered to Australian residents across its jurisdictions is measured in accordance with a national performance indicator framework.¹¹ The measures therein address response times, patient satisfaction, workforce and a small number of clinical conditions, including cardiac arrest and pain management. These measures do not reflect the breadth and complexity of care and services that modern ambulance services provide. As such, there is a need for a contemporary and meaningful suite of prehospital care QIs that Australian ambulance services could draw from for national reporting, benchmarking and quality improvement purposes.

This study formed part of a larger research project aimed at developing and testing prehospital care QIs for the Australian setting (www.aspireproject.net).¹⁷ An initial scoping review was conducted in accordance with JBI methodology to locate, examine and describe the international literature on indicators used to measure prehospital care quality.¹⁶ Subsequently, a modified RAND/UCLA appropriateness methods (RAM) was undertaken to aggregate the QIs identified in the scoping review, develop a suite of prehospital care QIs, and to assess the QIs for clarity and validity.¹⁸ Preparatory work for the expert consensus process included streamlined evidence syntheses guided by the JBI approach for rapid reviews and evidence summaries.^19,20

The current study set out with the aim to gain insight into the acceptability of a predetermined suite of 84 scientifically valid prehospital care QIs from the perspective of paramedics and ambulance service managers.

Methods

The methods applied in this study and other parts of the project were specified in advance in a protocol.¹⁷ Data collection for this study was commenced in February 2021 and completed in August 2021. The authors have no conflict of interest to declare, and the research was conducted independently and not at the behest of any regulatory body or ambulance service.

Preceding work

The 84 proposed QIs stemmed from previous published studies of the project, namely an initial scoping review,¹⁶ and a subsequent evidence-informed expert consensus process.¹⁸ In preparation for the consensus process, QIs identified in the scoping review were aggregated and systematically prepared within clinical and non-clinical domains, and a structure/process/outcome and access/safety/effectiveness taxonomy as summarised in Tables 1 and 2. The combination of best available evidence and expert consensus was used to identify existing QIs and to develop new ones to create a suite deemed clear and valid for the measurement of Australian prehospital care quality. The 84 valid QIs of the suite are listed in Supplementary Appendix A.

Table 1.

Categorisation of quality indicators into domains and sub-domains.

Domain A: Organisational/System	Domain B: Clinical
A.1. General Time IntervalsA.2. Patient SafetyA.3. Patient Experience and SatisfactionA.4. Communication and DispatchA.5. Resources and Resource ManagementA.6. Paramedic Health and SafetyA.7. Training Education and ResearchA.8. Other (Organisational/ System)	B.1. Airway Management, Ventilation and Oxygen TherapyB.2. Out-of-Hospital Cardiac Arrest (OHCA)B.3. Acute Coronary SyndromeB.4. StrokeB.5. AsthmaB.6. TraumaB.7. SeizuresB.8. HypoglycaemiaB.9. Pain ManagementB.10. Other (Clinical)

Table 2.

Dimensions of quality (adapted from Campbell et al.²¹ and Owen²²).

Structure	Process	Outcome	Dimension
AvailabilityAccessibilityEquity	AvailabilityAccessibilityContinuity/SustainabilityEquityTimeliness	Health StatusUser Evaluation (includes Acceptability)	Access
Safety (Provider and Patient)	Safety (Provider and Patient)	Absence of HarmProvider EvaluationUser Evaluation (includes Acceptability)	Safety
AppropriatenessCapabilityClinical EffectivenessCost-effectivenessEfficiencyEquity	AppropriatenessCapabilityCaringClinical EffectivenessContinuity/SustainabilityCost-effectivenessEfficiencyEquityInterpersonal effectivenessResponsivenessPatient-centerednessWell-led	Health StatusFinancial EvaluationUser Evaluation (includes Acceptability)	Effectiveness

Study design and setting

A two-staged explanatory sequential mixed methods research design was adopted.^23,24 Inquiries within the two stages of the study were guided by appropriate research paradigms depending on the specific objective of each stage. A postpositivist stance was taken in the initial quantitative stage followed by a constructivist stance in the subsequent qualitative stage.²³ Thus, researchers focused on the research questions and incorporated

plurality of methods to answer them.²⁵ Using Crotty's conceptualisation,²⁶ the study was further positioned within social science theories informed by reviews and frameworks of acceptability as a criterion for evaluating performance measures.^10,27,28 In Stage 1, an online survey was conducted to collect quantitative data on the acceptability of the 84 QIs. In Stage 2, online one-to-one semi-structured interviews were performed aimed at qualitatively explaining what makes QIs acceptable or unacceptable. Although results of the quantitative and qualitative aspects were integrated, the qualitative stage constituted the core of the research. Integration occurred at the conceptualisation of the study by planning an explanatory sequential design. During the research, integration was achieved through linking data collection and analysis. This was done by connecting through sampling, building by considering the results of the survey during the interviews, and merging the two datasets for analysis.^23,29 Integration through narrative was applied using a contiguous approach in reporting the results of the two stages, followed by weaving in the discussion.^29,30 The study was conducted in Australia.

Participants and recruitment

The target population for this study was comprised of paramedics and directors, managers, or supervisors who have worked on quality improvement projects from any Australian ambulance service. Recruitment involved website and email advertisement by the Australasian College of Paramedicine (ACP), the peak professional organisation with a membership of around 8,000 paramedics and paramedic students,³¹ followed by numerous social media posts on Twitter, LinkedIn and Facebook over a four-week period. At the end of the survey, participants of Stage 1 could express their interest in participating in Stage 2. The sample for Stage 2 was purposively selected and aimed at an even representation of demographic criteria by inviting survey participants to the subsequent interviews.

Stage 1 data collection

Participants were asked to anonymously complete an online survey (designed on Qualtrics; Qualtrics, Provo, Utah, USA). Since there was no existing survey that met the needs, the survey was purpose-built. The survey collected basic demographic data and then asked participants to answer the following question for each QI using a five-point numerical rating scale (1 = very unacceptable, 2 = unacceptable, 3 = neutral, 4 = acceptable, 5 = very acceptable): How acceptable is it to assess the quality of your patient care or the quality of your ambulance service based on data collected using this quality indicator? ‘Neutral’ signalled ambivalence in the participant's consideration of the acceptability or unacceptability of a QI. Acceptability was defined as the degree to which something is tolerated, allowed, or okay to use. Considering the brevity and simple survey flow as well as the earlier work optimising clarity of the QIs, piloting was not deemed necessary. Survey settings prevented multiple submissions by individuals. Based on the Australian registered paramedic population of approximately 17,000, using a sample size estimation with a confidence level of 95% and, for practical reasons, accepting a margin of error of 8%, an ideal sample size of 149 was pursued.

Stage 1 data analysis

Quantitative data analysis was performed using Microsoft Excel V16 (Microsoft, Richmond, Washington, USA) and IBM SPSS Statistics V27 (IBM, Armonk, New York, USA). Descriptive statistics were completed to summarise all survey items. For each QI, central tendency of acceptability ratings was evaluated using the median. The five-point rating scale was assumed to represent a continuous variable rather than five discrete categories, and medians were calculated accordingly.^32,33 Diverging stacked bar charts were created to visualise distributions. Explicit acceptability and unacceptability were calculated to be expressed as percentages by combining ratings of four (acceptable) and five (very acceptable), and two (unacceptable) and one (very unacceptable), respectively. Kruskal–Wallis tests (KWt) were conducted to examine the differences on medians according to QI types (structure, process and outcome) and quality dimensions (access, safety and effectiveness). A p value of <0.05 was considered statistically significant. Finally, the median of medians (as the preferred measure of centre in light of the distribution of medians) was identified for the entire suite, its two domains, as well as subsets of QIs in accordance with the project's classification system.

Stage 2 data collection

Guided by methods described by DiCicco-Bloom and Crabtree,³⁴ an interview guide containing a priori questions and topics/questions informed by the results of Stage 1 was developed. The interview guide facilitated a flexible interview protocol and was supplemented by follow-up questions, probes and comments.^35–37 The open-ended questions aimed at eliciting explanations for what makes QIs acceptable or unacceptable to participants. Participants were asked specifically about QIs which were rated relatively low or high in Stage 1. To ensure diversity and to optimise credibility of results, maximum variation sampling was used from the pool of volunteering participants.³⁸ Interviews, with an intended length of approximately 30 min, were conducted in English by the principal investigator (RP) using audio call on Zoom (Zoom Video Communications Inc., San Jose, CA, USA) and recorded for transcription. During the interviews, member checking was performed through the interviewer restating or summarising information and then questioning the participant to determine validity.³⁵ Field notes were taken during and after each interview. Data were collected until saturation, defined as the point at which no new information was observed.^39,40 One additional interview was conducted for assurance confirming no new themes were observed in the data.⁴¹ Trustworthiness of the findings was enhanced by data triangulation of recorded interviews, transcripts and field notes.²³

Stage 2 data analysis

Using NVivo 12 (QRS International, Doncaster, Australia), the transcripts were analysed by conducting thematic analysis as outlined by Braun and Clarke.⁴² Analysis was performed by one researcher (RP) using an inductive and semantic approach. The triangulation which was performed to ensure credibility also enabled content familiarisation (phase 1). Data were disassembled through coding (phase 2) by systematically identifying aspects of interest and collating those relevant to each other. Codes were then reassembled by placing them into context with each other to create potential themes and, where appropriate sub-themes improving structure to relatively large and more complex themes (phase 3). Themes were reviewed developing a thematic ‘map’ of the analysis within NVivo (phase 4). Ongoing analysis facilitated the refinement of the themes and eventually the allocations of definitions and names for each theme and sub-theme (phase 5). Finally, compelling extracts were selected to perform conclusive analysis and relate the analysis to the research question on acceptability (phase 6).⁴² During each of the six phases, means of establishing trustworthiness as described by Nowell and colleagues⁴³ were applied.

Results

Stage 1

Of 44 responses to the survey, 36 (82%) were complete and thus utilisable for data analysis. Participant demographics are detailed in Table 3. Participants were predominantly male (61.1%) and relatively young (72.2% aged between 25 and 44 years). Half of all participants had a paramedic specialist qualification, and more than half (58.3%) were from New South Wales (NSW). Experience ranged from less than five years to more than 24 years. Two-thirds of participants worked primarily as clinicians, whereas the other third chiefly had managerial responsibilities.

Table 3.

Participant demographics.

	Stage 1		Stage 2
	n	%	n	%
Total participants	36	100	9	100
Gender
Male	22	61.1	6	66.7
Female	14	38.9	3	33.3
Age range (years)
18–24	0	0.0	0	0.0
25–34	15	41.7	2	22.2
35–44	11	30.6	5	55.6
45–54	8	22.2	2	22.2
55–64	2	5.6	0	0.0
>64	0	0.0	0	0.0
Qualification
Paramedic Trainee	2	5.6	0	0.0
Qualified Paramedic	16	44.4	4	44.4
Paramedic Specialist (ICP, CCP, ECP, etc.)	18	50.0	5	55.6
State/Territory
Australian Capital Territory	1	2.8	0	0.0
New South Wales	21	58.3	5	55.6
Northern Territory	2	5.6	0	0.0
Queensland	4	11.1	1	11.1
South Australia	3	8.3	1	11.1
Tasmania	0	0.0	0	0.0
Victoria	2	8.3	1	11.1
Western Australia	3	5.6	1	11.1
Experience (years full-time)
<5 years	6	16.7	0	0.0
5–9 years	9	25.0	5	55.6
10–14 years	8	22.2	0	0.0
15–19 years	3	8.3	1	11.1
20–24 years	5	13.9	3	33.3
>24 years	5	13.9	0	0.0
Primary role
Clinician	24	66.7	5	55.6
Director, manager, supervisor	12	33.3	4	44.4

CCP, Critical Care Paramedic; ECP, Extended Care Paramedic; ICP, Intensive Care Paramedic.

Overall, the acceptability of all QIs in the suite was rated highly. Figure 1 shows the left-skewed distribution of acceptability rating medians for all QIs and further results for all QIs of the suite can be found in Supplementary Appendix A. Table 4 highlights those QIs which received relatively low acceptability ratings. Kruskal–Wallis tests showed that there was a statistically significant difference in medians between the different QI types (structural, process and outcome) (KWt(2) = 13.260; p = 0.001), however, no significant difference in medians between QIs addressing the three quality dimensions (access, safety and effectiveness) was found (KWt(2) = 0.437; p = 0.8).

Figure 1.

Distribution of median acceptability ratings for all 84 Qis.

Table 4.

Stage 1 results for quality indicators which were rated relatively low.

Quality Indicator (Type; Quality dimension/s)	Explicitly Unacceptable	Explicitly Acceptable	Median
QI-A.3.6. In a patient satisfaction survey, a patient reports that they were very satisfied or satisfied with the ambulance services they received in the previous 12 months. (Outcome; Access, Safety, Effectiveness)	13.9%	61.1%	3.81
QI-B.2.1. An ambulance arrives at an OHCA patient within 4 min of the 000-call. (Process; Access)	30.6%	52.8%	3.61
QI-B.5.1. A suspected acute asthma patient has their PEF measured prior to nebulisation unless they are unable to perform the test. (Process; Effectiveness)	33.3%	47.2%	3.36
QI-A.6.2. The ambulance service schedules paramedics to work shifts shorter than 12 h in duration. (Process; Safety)	38.9%	38.9%	3.00

very unacceptable; unacceptable; neutral; acceptable; very acceptable.

OHCA Out-of-Hospital Cardiac Arrest; PEF Peak Expiratory Flow.

The median of medians for all QIs in the suite was 4.60 (IQR 0.32). The medians of medians for QIs within the organisational/system and the clinical domains were identical and equal to that of all QIs (4.60; IQR 0.33). Medians of medians for structural, process and outcome indicators were 4.60 (IQR 0.11), 4.62 (IQR 0.36) and 4.15 (IQR 0.45), respectively. For QIs addressing access, safety and effectiveness, the medians of medians were 4.58 (IQR 0.38), 4.60 (IQR 0.17) and 4.60 (IQR 0.32), respectively.

Stage 2

Data saturation was achieved after eight interviews. The additional ninth interview yielded no new information. Six of the nine interviewees were clinicians and four were managers/supervisors. Participant demographics are summarised in Table 3 and individual attributes can be found in Supplementary Appendix B. The mean interview duration was 29 min, ranging from 15 to 45 min. Overall, five themes and six sub-themes were created (Table 5).

Table 5.

Themes and sub-themes

Themes	Sub-themes
Key Characteristics of QIs: Possessing certain key characteristics makes QIs more acceptable.	Clarity: Being clear makes QIs more acceptable. Evidence-based: Being based on best available evidence makes QIs more acceptable. Practicality: To be acceptable, QIs need to be describe aspects that are realistically achievable. Meaningfulness: Describing aspects of care that are meaningful makes QIs more acceptable.
2. Patient Satisfaction: QIs which describe aspects of patient satisfaction are less acceptable.	Lesser Priority: Patient satisfaction is of lesser priority, and thus related QIs are less acceptable. Proxy Measures: Patient satisfaction is a proxy measure of quality, and thus associated QIs are less acceptable.
3. Outcome Indicators: QIs that describe desirable patient outcomes are generally better. However, they often are less specific prehospital care QIs and thus less acceptable.	NA
4. Time Intervals: The acceptability of QIs which include time intervals depends on how specific they are about time-sensitive patients and treatments.	NA
5. Professional Values and Qualities: Linking QIs to professional values and qualities is challenging.	NA

Theme 1: Characteristics of QIs

To increase the level of acceptability, a QI needed to possess certain characteristics which made it suitable to be implemented for its intended use (i.e., the measurement of quality). These characteristics included clarity, being evidence-based, practicality and meaningfulness. The more a QI was perceived to have these desirable characteristics, the more acceptable it was to participants. Vice versa, QIs which lacked these attributes, were less acceptable.

Sub-theme 1a: Clarity

QIs which were ambiguous were less acceptable to participants. QIs needed to be detailed enough so that what would be assessed could be attributed exactly to that QI. Correspondingly, it needed to be clear if there are situations in which the measurement related to the QI should not be performed.

…there may be occasions where oxygen would be appropriate, slash acceptable […]. But I

suppose the crunch of that [QI] would be “unless specifically indicated’, which is quite generic and probably quite subjective. (Participant #6)

This participating manager explained why they rated the acceptability of QI-B.1.3. relatively low. The phrases “unless specifically indicated” was considered vague and insufficient. For this QI to be more acceptable, it needed to detail certain patient variables to describe the exact clinical scenario when the treatment of interest should be administered.

Sub-theme 1b: Evidence-based

When asked what makes a QI acceptable, a common response from participants was that they should be based on best available evidence. Such QIs were more valid and thus more acceptable to participants. On the other hand, some QIs which were seen as being ill-supported by evidence was considered less acceptable.

[QIs] should be evidence informed. There's no point having quality indicators that

say, “do X”, if×is way out of date. (Participant #3)

I would say that having an evidence base to support methodologies is pretty important.

Because what you want to use is validated approaches that give some credibility to whatever you report. (Participant #8)

Sub-theme 1c: Practicality

Practicality influenced the level of acceptability. Especially participating clinicians considered those QIs which in their eyes described realistically achievable and contextually viable aspects of prehospital care to be more acceptable. Often this was described in terms of the holistic service delivery rather than sub-sets of patient encounters. Thus, there was an element of equity, too.

Quality indicators need to ensure that quality healthcare is delivered, but that it's

attainable by all members of the workforce. (Participant #1)

One participant talked about a major trauma patient they recently treated and how QI-B.6.4. would not have been met and how this QI details an unrealistic practice.

But all of this takes way more than 10 min. It's totally unrealistic to be suggesting that

we're going to be on the road on the way to definitive care in 10 min. It's just not happening. You know what I mean? I don't even see how it can. (Participant #4)

Some participants identified a potential risk of ‘pleasing the QI’ when impractical QIs are implemented, meaning that there is a conflict between what the QI describes and what is in the best interest of the patient. Paramedics may reluctantly provide patient care that is in line with the QI but would see true quality of care being compromised. This made participants comment on flexibility in the interpretation of measurement data and the associated differentiation between warranted and unwarranted variation. Variation from what a QI dictates was considered to be warranted when it is in the best interest of patients’ clinical needs or preferences.

I think it also needs to be flexible. So you need to understand why, in some instances, there's

deviations from the quality attributes. So if something's happening, sort of investigate why it's happening and then feed that back. (Participant #7)

Sub-theme 1d: Meaningfulness

An acceptable QI described aspects of care that were meaningful to participants. In other words, they needed to describe aspects of prehospital care that conform to the individual participant's ideology of quality in this context. Of the many dimensions of quality, participants placed most emphasis on effectiveness. QIs that focus on the impact of prehospital care provided to patients and communities were considered more acceptable.

Safety and patient-centredness also featured as desirable attributes of quality.

It has to be something that's generally involved in patient outcomes as best as possible, and

that's multi-faceted. So that's not just response time to getting there, but it's actually what we're able to do for the patient once we are there. (Participant #2)

One of the managers commented on the purpose of measurement often being lost in large organisations and questioning the meaningfulness of the collected and analysed data.

Sometimes I think when you get into big organizations […] we ask big questions, and we ask

for big packets of data. But we're not really sure what we're asking, why we're asking for it, and what its application is in terms of the strategic information and overview and what is it going to drive us to change or verify. (Participant #8)

Theme 2: Patient satisfaction

QIs that describe aspects of patient experience and satisfaction was seen by the participants as less important and limited in their validity to be used as measures of quality.

Sub-theme 2a: Lesser priority

The measurement of patient experience and satisfaction was seen as less of a priority compared to other aspects of service and care. This explains the relatively low acceptability rating of QIs describing aspects of patient experience and satisfaction.

In our environment, I think to have an expectation that you have this amazing customer

service experience, amazing clinical care, and amazing outcomes. People are fallible and I don't think we can provide that every time. (Participant #1)

Sub-theme 2b: Proxy measures

QIs describing aspects of patient satisfaction were seen as limited in their validity as measures of quality of prehospital care. Patient satisfaction metrics were seen to represent the patient's subjective contentment with the service, a distinct aspect of care. Patient satisfaction QIs were considered important patient-centred measures, but participants did not think that they should be used as a proxy for overall quality of prehospital care.

Sometimes we're unable to explain our thought processes to our patients, to a level that they

understand, […]. I found that quality indicator maybe is very dependent on the patient. I don't know if it would work as well as the others. (Participant #3)

Theme 3: Outcome indicators

Generally, the importance of patient outcomes and associated measurement was well recognised by participants. Structure and process type QIs were seen as more acceptable when they aligned with outcomes of interest. Nevertheless, some clinicians raised concerns about outcome indicators and their sensitivity to differences in prehospital quality of care. Prehospital care was described as a brief initial part of a much more extensive care pathway, especially in critically ill or injured patients. Outcome measurement at a distant point in that pathway may have limited ability to determine prehospital care quality.

I think, um, as far as determining quality, it's about, you know, a set of outcomes. […]

[QIs] have to be tangible and linked to an outcome. (Participant #1)

Well, 30 days after the event, who knows what's happened to them in hospital? I was unsure

about how that really linked to prehospital quality of care because it was a little bit like, well after 30 days they could have had exceptional ICU care, or they could have been in [hospital name] and it could have been [obscenity]. So their outcome could be a poor one, but there's no way that that relates to the prehospital quality of care that they received. (Participant #4)

Theme 4: Time intervals

When asked about indicators which involve general time intervals, participants considered these to be less acceptable. QIs which were specific about time-sensitive patient cohorts (i.e., those with conditions requiring timely interventions not available in the prehospital setting such as endovascular thrombectomy, damage control surgery, or percutaneous coronary intervention) were seen as more acceptable. It was important to participants that any QI which included time-intervals was specific about critical interventions rather than less meaningful aspects, such as arrival on scene or non-specific transport destinations. Any QI with time frames needed to be achievable and contextual, reiterating the importance of practicality.

There's no robust evidence that demonstrates that arriving at a scene within eight minutes of

being dispatched improves anything. There's no evidence to support any of that. (Participant #4)

I consider it important for certain things like say for chest pain or stroke symptoms, I

consider it important, where time is a factor when it comes to treatment. (Participant #3)

I feel like just saying that we need to get to every job within X time or get to hospital

within Y time, is no longer a good indicator of whether we've provided the right care […]. (Participant #8)

Theme 5: Professional values and qualities

As part of the a priori interview questions, participants were asked how the proposed QIs aligned with their professional values and qualities. Participants found it challenging to comment on this. Nevertheless, most participants said that the suite of QIs connected to what they believed to be their professional values and qualities.

‘I don't quite know how to answer that, I'll be honest. But I think that I would say that they

mostly aligned to most of the stuff that we want to do and that we should be aspiring to do’. (Participant #8)

‘Well, it's quite a difficult one to answer. To me, it's about providing good patient

care, what's safe and effective and patient focused. And I think anything that aligns with what's best practice definitely then aligns with my own values’. (Participant #9)

Discussion

Overall, participants found almost all QIs in the proposed suite to be acceptable. If a cut-off median score of 3.5 or greater in the ratings was applied, the initial list of 84 QIs would be reduced by only two. Nevertheless, besides commenting on desirable factors, participants also described aspects of the QIs that negatively affected the level of acceptability.

For participants in this study, the acceptability of the proposed QIs was dependent on the perceived presence of other key characteristics. When participants thought that a QI lacked clarity or when they believed a QI was poorly supported by evidence, they rated this QI as less acceptable. The need for a QI to possess clarity (a proxy for content validity) and be supported by evidence suggests a positive association between how acceptable QIs are to healthcare providers and scientific acceptability. Scientific acceptability addresses, at least in part, the basic measurement principle of validity.⁴⁴ Whilst the concept of validity is often applied to the results of measurement, it must also be considered within the elements of QIs and hence their development. In QI development, validity refers to the correctness of the QI as compared to scientifically credible sources.^5,44 Unsurprisingly, the findings of this study suggest that QIs which are developed systematically on the premise of being based on high-quality evidence, or a combination of best-available evidence and expert opinion when high-quality evidence is scarce or absent, are more likely to be accepted by healthcare providers. The link between undisputable evidence and acceptability provides plausible explanation for some of the highest ratings, e.g., QI-A.4.3. or QI-B.1.8. Since all QIs in this study were previously validated, this notion may also explain the high acceptability ratings in general.

Considerable commentary emerged about the practicality of what QIs describe. The acceptability of some QIs was rated relatively low by some participants because they felt that the described aspects were unrealistic, e.g., QI-A.6.2. and QI-B.6.4. Undoubtedly, this is one of this study's most important findings. Involving those who will be assessed and those who will conduct the measurement, analysis and resultant decision-making can provide a useful reality check for whether the QI is practical and sensible. Primarily this enables refinement of QIs and serves as a catalyst for their effective application. In other words, assessing the acceptability of QIs may increase their acceptability and hence successful use.⁴⁵

Linked to practicality of what QIs describe, participants also highlighted the importance of flexibility in QI application. The concept of variation may be considered in somewhat different ways. In performance measurement, variation often refers to changes in the data over time, its interpretation being one of the cornerstones of improvement science.⁴⁶ However, the variation that interview participants referred to is best described as a difference in healthcare processes, compared to peers or to a gold standard such as an evidence-based guideline recommendation,⁴⁷ or a QI. Variation is not automatically an indicator of poor quality. In fact, to some degree, variation should always exist because patients are unique and care should be responsive to differences between patients.^48,49 Participants raised concerns about inflexibility of QI application, and sensitivity to patient characteristics or situational demands not being recognised as warranted variation. Many participants highlighted that this may lead to ‘pleasing the QI’. In other words, paramedics providing prehospital care that aligns with applicable QIs even when this compromises patient-centredness or other genuine aspects of quality.

Lastly within the ‘key characteristics of QIs’ theme, meaningfulness featured strongly in the interviews. For any indicator to be meaningful, it must have a relationship to the underlying phenomenon it is intending to signal.⁵⁰ Therefore, a prehospital care quality indicator must describe aspects of good patient care delivered by ambulance services. When theory or scientific evidence is robust, this link is clear.⁵¹ However, in developing healthcare disciplines with a limited evidence base such as paramedicine, there may be dispute over the strength of a relationship between the proposed indicator and the underlying phenomenon to be measured. In the absence of an agreed definition of prehospital care quality, the acceptability of associated indicators will inevitably vary. This provides an explanation of the wide range of ratings for some of the QIs, e.g., QI-A.6.2. or QI-B.2.7. More importantly, it highlights the need for consensus on a definition of prehospital care quality as it would contribute to more consistent acceptance of aligned QIs and ultimately their strength in affecting meaningful improvement.

The first theme and its sub-themes reflect findings of similar studies investigating the acceptability of QIs in other healthcare disciplines. Burges and colleagues,⁵² for example, found that QIs which were evidence-based and clearly worded were more acceptable to primary healthcare providers (family physicians, nurses and nurse practitioners) participating in their Delphi study. Increasingly QIs are described in terms of being fit-for-purpose and fit-for-use, together contributing to their actionability. A recent multiphase qualitative analysis exploring the concept of actionability of QIs. Barbazza, Klazinga and Kringos describe three clusters within which a QI's fitness for use can be appraised: methodological, contextual and managerial.⁵³ In particular, the methodological considerations resonate with the findings of this study supporting the idea that a QI which is systematically developed with careful considerations of key characteristics will be more acceptable to prehospital care clinicians and managers and ultimately possess more potential to facilitate improvement.

Different perspectives on health care quality often result in different expectations and thus different indicators of quality. As illustrated in sub-theme 1d, healthcare providers frequently view quality through clinical effectiveness and associated outcomes. Whilst outcomes are important to patients too, they frequently place extensive value on the emotional or interpersonal aspects of care.⁵⁴ This might be especially true in settings like prehospital care where noticeable outcomes are seldomly reached due to short patient contact times. As a result, participants considered QIs which described aspects of patient satisfaction to be of lesser priority and less valid as QIs of prehospital care. This does not mean that participants disregarded patient values or that QIs describing aspects of patient experience and satisfaction were unacceptable. It means that perspective matters and that the development of a symmetrical suite of QIs balancing different perspectives on quality will need to involve patients.⁵⁵ Whilst patient satisfaction is widely measured in healthcare, our finding indicates that it might still be a poorly understood concept among paramedics. This aligns with results of recent systematic reviews on the concept and determinates of patient satisfaction (irrespective of healthcare discipline) suggesting that there is a need for further research aiming to define patient satisfaction and how to validly utilise it as an indicator of healthcare quality.^54,56

In the evaluation of quality of health care, structural, process and outcome indicators all have advantages and disadvantages.⁵⁷ The perceived level of acceptability of outcome QIs was somewhat conflicted by the advantages and disadvantages of this type of indicator. Participants realised that outcome QIs are beneficial since they facilitate measurement of something that is important in its own right. However, since most outcome measurement will occur sometime after the brief prehospital care phase, it is reflective of all aspects of healthcare, not only that provided by paramedics. Outcome measurement is also influenced by variables other than healthcare processes, e.g., patient characteristics. Therefore, although generally recognised as important types of QIs, participants expressed concern about the attributability of outcome indicators in evaluating prehospital care quality. This explains the somewhat wider distribution in the acceptability ratings of QIs-B.2.8 to 10 and the statistically significant difference in medians between the different QI types. The problem of attribution in outcome measurement is not unique to prehospital care though. Attributing outcomes to the care and services provided by organisations or individuals in most healthcare discipline is often challenging due to the many factors that can influence a particular outcome.⁵⁸ Furthermore, the fact that outcomes may need a long time to manifest was recognised by Donabedian in his early conceptualisation of the various types of indicators.⁵⁹ Thus, similar concerns about the acceptability of outcome indicators were raise by primary healthcare providers in the study by Burge et al. where some participants felt that they had no power to influence outcomes and thus they considered them poor (and hence less acceptable) indicators of the quality of the care they provided.⁵²

Timeliness was considered to be an important attribute of prehospital care quality. Participants agreed that in time-sensitive patients, such as cardiac arrest, stroke, or major trauma, timely access to healthcare contributes to desirable health outcomes, e.g., QI-B.2.3., QI-B.3.8. and QI-B.3.9.addl. However, participants reiterated what has been debated within the paramedicine discipline for some time – there is little evidence to support the generic measurement of response times as an indicator of prehospital care quality.⁶⁰ It is worth noting at this point that in this project, indicators detailing general time intervals such as response time, time on scene, or turnaround time, were all deemed not valid in the preceding study.¹⁸ Advances in ambulance deployment modelling and call triaging, and a systematically developed suite of QIs, should contribute to more sustainable performance and meaningful measurement of timely access to health care.

Participants found it difficult to comment on how the proposed QIs connected with their professional values and qualities. Similar to other registered healthcare professions, paramedicine in Australia is regulated by its own regulatory authority, the Paramedicine Board of Australia. The code of conduct for registered health practitioners was developed by most of the 15 National Boards. It states that “while individual practitioners have their own personal beliefs and values, there are certain professional values on which all practitioners are expected to base their practice”.⁶¹ ^{(p. 6)} The code describes a framework for the provision of appropriate, effective, and ethical health care. Thus, there should be a fundamental link between guidance on how to provide high-quality patient care and indicators thereof. Although hesitant, participants seemed to consider their professional values and qualities by reflecting on what is meaningful prehospital care and considered the suite of QIs to be in line. The alignment of QIs with professional values (and vice versa) is critical because the success of systems and processes to monitor, manage and regulate the quality of care provided to patients depends significantly on the values and behaviours of staff working throughout the system.⁶²

Limitations

The recruitment strategy includes a risk of selection bias. A range of strategies were considered and despite this limitation, our strategy still represented the optimal approach to attract participants. The formally calculated sample size (n = 149) for the survey was not reached. Therefore, Stage 1 is underpowered and at risk of type 2 error. Maintaining a confidence level of 95%, the sample size of 36 leads to a 16% margin of error signalling low confidence in the results of this study being representative of the Australian paramedic population. The results remain of value but should be considered as hypothesis generating rather than definitive. Limitations inherent to Likert-type questions may be present, namely central tendency bias, acquiescence bias, and social desirability bias; the latter two being more likely in light of the results. A “neutral” response could mean that participants were undecided on how to answer or that, despite previous work on optimising clarity as detailed in the Introduction, the particular QI was not clear to participants. Participants from NSW were overrepresented in Stage 1 of the study. Despite maximum variation sampling in Stage 2, the overrepresentation remained. Although semi-structured interview guides were used, the principal investigator's understanding and interpretation of the data may have potentially introduced confirmation bias. Furthermore, since the principal investigator conducted the interviews, he was not blinded to the study objectives, thereby potentially introducing bias. Determining the point of thematic saturation was based only on the information that was found, and on the judgement and experience of the researchers. Whilst multiple best practices for rigor in qualitative research were followed, transcripts were not returned to participants for checking of themes and sub-themes. Further research is needed, which was beyond the scope of this study, to investigate how acceptable the proposed QIs are to patients and communities.

Conclusion

The findings of this study provide insight into how acceptable the proposed suite of QIs is to paramedics. More specifically, the results suggest that 82 of the 84 QIs may be acceptable to prehospital care providers. Through Stage 2 of this study, several attributes have been identified which could be transferrable to other prehospital care QI development initiatives. Through careful consideration of these, QI developers might ensure acceptability of their QIs from the start of the development process. Within this project, future research should evaluate the QIs’ feasibility and reliability, investigate how acceptable they are to patients and communities, and look at ways to effectively implement the proposed suite of prehospital care QIs.

Supplemental Material

sj-docx-1-pam-10.1177_27536386231158390 - Supplemental material for Acceptability of Australian prehospital care quality indicators: An explanatory sequential mixed methods study

Supplemental material, sj-docx-1-pam-10.1177_27536386231158390 for Acceptability of Australian prehospital care quality indicators: An explanatory sequential mixed methods study by Robin Pap, Matthew Stephenson, Paul Simpson and Craig Lockwood in Paramedicine

Supplemental Material

sj-docx-2-pam-10.1177_27536386231158390 - Supplemental material for Acceptability of Australian prehospital care quality indicators: An explanatory sequential mixed methods study

Supplemental material, sj-docx-2-pam-10.1177_27536386231158390 for Acceptability of Australian prehospital care quality indicators: An explanatory sequential mixed methods study by Robin Pap, Matthew Stephenson, Paul Simpson and Craig Lockwood in Paramedicine

Footnotes

Acknowledgments

This study,and the research project it forms part of,receives support through the provision of an Australian Government Research Training Program Scholarship.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research,authorship,and/or publication of this article: Robin Pap is an Associate Editor of Paramedicine and Paul Simpson is the Editor-in-Chief of Paramedicine .

Funding

This study was supported by a research grant from the Australasian College of Paramedicine (ACP).

Ethics approval and consent to participate

Ethical approval was granted by the University of Adelaide Human Research Ethics Committee (H-2017-157). All methods were carried out in accordance with the Australian Code for the Responsible Conduct of Research. Informed consent was obtained from all participants.

Consent for publication

Not applicable.

Availability of data and materials

The complete data that support the findings of this study are available from the corresponding author,but restrictions apply to the availability of these data and so are not publicly available. Data are however available from the corresponding author upon reasonable request and with permission of participants.

Authors’ contributions

RP conceived the project,designed the study,and obtained ethics approval and research funding. MS,PS and CL supervised the conduct of the study. RP build the survey,undertook recruitment,conducted the interviews and performed data analyses. RP wrote the manuscript. All authors contributed intellectually and reviewed it. RP takes responsibility for the paper as a whole.

ORCID iD

Robin Pap

Supplemental material

Supplemental material for this article is available online.

Correction (September 2023):

This article has been updated to disclose the Declaration of conflicting interests statement.

References

Mainz

. Defining and classifying clinical indicators for quality improvement. International Journal for Quality in Health Care 2003; 15: 523–30.

Perla

Provost

Parry

. Seven propositions of the science of improvement: exploring foundations. Qual Manag Health Care 2013; 22: 170–86.

Marshall

Pronovost

Dixon-Woods

. Promotion of improvement as a science. The Lancet 2013; 381: 419–21.

Kringos

Groene

Johnsen

. Training the first generation of health care performance intelligence professionals in Europe and Canada. Academic Medicine 2019; 94: 747–8.

Mainz

. Developing evidence-based clinical indicators: a state of the art methods primer. International Journal for Quality in Health Care 2003; 15: i5–11.

Campbell

Kontopantelis

Hannon

, et al. Framework and indicator testing protocol for developing and piloting quality indicators for the UK quality and outcomes framework. BMC Fam Pract 2011; 12: 85.

Schang

Blotenberg

Boywitt

. What makes a good quality indicator set? A systematic review of criteria. Int J Qual Health Care 2021; 33: 1–10.

Campbell

Braspenning

Hutchingson

, et al. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care 2002; 11: 358–64.

Jones

Shepherd

Wells

, et al. Review article: what makes a good healthcare quality indicator? A systematic review and validation study. Emerg Med Australas 2014; 26: 113–24.

10.

Sekhon

Cartwright

Francis

. Acceptability of healthcare interventions: An overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017; 17: 1–13.

11.

Australian Government Productivity Commission - Steering Committee for the Review of Government Service Provision. Report on Government Services 2022, Ambulance Services [Internet]. Canberra; 2022. Available from: https://www.pc.gov.au/research/ongoing/report-on-government-services.

12.

Harrison

. Ambulance Services in Australia - Healthy growth: Population growth has increased demand for ambulance services. Melbourne; 2019.

13.

Braithwaite

Churruca

Ellis

, et al. Complexity science in healthcare. Sydney: Macquarie University, 2017, pp.118.

14.

Provost

Murray

. The health care data guide: learning from data for improvement. 1st ed. Hoboken, NJ: Wiley, 2011.

15.

Quentin

Partanen

Brownwood

, et al. Measuring healthcare quality. In: Busse

Klazinga

Panteli

Quentin

(eds) Improving healthcare quality in Europe Characteristics, effectiveness and implementation of different strategies [Internet]. Copenhagen (Denmark): European Observatory on Health Systems and Policies, 2019, pp.420. Available from: https://www.ncbi.nlm.nih.gov/books/NBK549260/

16.

Pap

Lockwood

Stephenson

, et al. Indicators to measure prehospital care quality: a scoping review. JBI Database System Rev Implement Rep 2018; 16: 2192–223.

17.

Pap

Lockwood

Stephenson

, et al. Development and testing of Australian prehospital care quality indicators: study protocol. BMJ Open 2020; 10: e038310.

18.

Pap

Lockwood

Stephenson

, et al. Development of prehospital care quality indicators for the Australian setting : a modified RAND / UCLA appropriateness method. Emergency Medicine Journal 2021; 0: 1–6.

19.

Pap

McKeown

Lockwood

, et al. Pelvic circumferential compression devices for prehospital management of suspected pelvic fractures: a rapid review and evidence summary for quality indicator evaluation. Scand J Trauma Resusc Emerg Med 2020; 28: 65.

20.

Munn

Lockwood

Moola

. The development and use of evidence summaries for point of care information systems: a streamlined rapid review approach. Worldviews Evid Based Nurs 2015; 12: 131–8.

21.

Campbell

Roland

Buetow

. Defining quality of care. Soc Sci Med 2000; 51: 1611–25.

22.

Owen

. The development and testing of indicators of prehospital care quality [dissertation]. [Manchester]. University of Manchester; 2010.

23.

Creswell

Plano Clark

. Designing and conducting mixed methods research. 2nd ed. Los Angeles: Sage Publications Inc., 2011, pp.457.

24.

Ivankova

Creswell

Stick

. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods 2006; 18: 3–20.

25.

Teddlie

Tashakkori

. Foundations of mixed methods research : integrating quantitative and qualitative techniques in the social and behavioral sciences. Thousand Oaks, CA: Sage Publications, 2009.

26.

Crotty

. The foundations of social research Meaning and perspective in the research process. London: Routledge, 1998.

27.

Levy

Williams

. The social context of performance appraisal: A review and framework for the future. J Manage 2004; 30: 881–905.

28.

Hedge

Teachout

. Exploring the concept of acceptability as a criterion for evaluating performance measures. Group Organ Manag 2000; 25: 22–44.

29.

Fetters

Curry

Creswell

. Achieving integration in mixed methods designs - Principles and practices. Health Serv Res 2013; 48: 2134–56.

30.

Creswell

Tashakkori

. Editorial: developing publishable mixed methods manuscripts. J Mix Methods Res 2007; 1: 107–11.

31.

Australasian College of Paramedicine. Australasian College of Paramedicine 2022 Annual Report. Umina Beach, 2022.

32.

Harpe

. How to analyze Likert and other rating scale data. Curr Pharm Teach Learn 2015; 7: 836–50.

33.

Sullivan

Artino

. Analyzing and Interpreting data from likert-type scales. J Grad Med Educ 2013; 5: 541–2.

34.

DiCicco-Bloom

Crabtree

. The qualitative research interview. Med Educ 2006; 40: 314–21.

35.

DeJonckheere

Vaughn

. Semistructured interviewing in primary care research: A balance of relationship and rigour. Fam Med Community Health 2019; 7: 1–8.

36.

Kallio

Pietilä

Johnson

, et al. Systematic methodological review: developing a framework for a qualitative semi-structured interview guide. J Adv Nurs 2016; 72: 2954–65.

37.

Britten

. Qualitative Interviews. In: Pope

Mays

(eds) Qualitative research in health care. 3rd ed. John Wiley & Sons, Incorporated, 2006.

38.

Teddlie

. Mixed methods sampling: a Typology with examples. J Mix Methods Res 2007; 1: 77–100.

39.

Guest

Bunce

Johnson

. How many interviews are enough?: An experiment with data saturation and variability. Field Methods 2006; 18: 59–82.

40.

Fusch

Ness

. Are we there yet? Data saturation in qualitative research. Qualitative Report 2015; 20: 1408–16.

41.

Guest

Namey

Chen

. A simple method to assess and report thematic saturation in qualitative research. PLoS One 2020; 15: 1–17.

42.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

43.

Nowell

Norris

White

, et al. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods 2017; 16: 1–13.

44.

Ferris

Amster

Dunton

, et al. Guidance for Measure Testing and Evaluating Scientific Acceptability of Measure Properties. 2011.

45.

Jones

Kwong

Warburton

. Quality improvement made simple: what everyone should know about health care quality improvement. 3rd ed. London: The Health Foundation, 2021, pp.67.

46.

Neuhauser

Provost

Bergman

, et al. The meaning of variation to healthcare managers, clinical and health-services researchers, and individual patients. BMJ Qual Saf 2011; 20.

47.

Australian Commission on Safety and Quality in Healthcare. Australian Atlas of Healthcare Variation. Sydney; 2021.

48.

Atsma

Elwyn

Westert

. Understanding unwarranted variation in clinical practice: A focus on network effects, reflective medicine and learning health systems. International Journal for Quality in Health Care 2020; 32: 271–4.

49.

Wennberg

. Unwarranted variations in healthcare delivery: Implications for academic medical centres. Br Med J 2002; 325: 961–4.

50.

Donabedian

. The quality of care: how can it be assessed? Journal of the American Medical A 1988; 260: 1743–8.

51.

Goldenberg

. Defining “quality of care” persuasively. Theor Med Bioeth 2012; 33: 243–61.

52.

Burge

Lawson

Putnam

. Assessing the acceptability of quality indicators and linkages to payment in primary care in Nova Scotia. Healthcare Policy 2011; 6: 72–87.

53.

Barbazza

Klazinga

Kringos

. Exploring the actionability of healthcare performance indicators for quality of care: A qualitative analysis of the literature, expert opinion and user experience. BMJ Qual Saf 2021: 1–11.

54.

Batbaatar

Dorjdagva

Luvsannyam

, et al. Determinants of patient satisfaction: A systematic review. Vol. 137, Perspectives in Public Health. Sage Publications Ltd, 2017, pp. 89–101.

55.

Berland

. Using the Johari Window to explore patient and provider perspectives. International Journal of Health Governance 2017; 22: 47–51.

56.

Batbaatar

Dorjdagva

Luvsannyam

, et al. Conceptualisation of patient satisfaction: A systematic narrative literature review. Vol. 135, Perspectives in Public Health. Sage Publications Ltd, 2015, pp. 243–50.

57.

Mant

. Process versus outcome indicators in the assessment of quality of health care. International Journal for Quality in Health Care 2001; 13: 475–80.

58.

Lilford

Mohammed

Spiegelhalter

, et al. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. The Lancet 2004; 363: 1147–54.

59.

Donabedian

. The Definition of Quality and Approaches to Its Assessment. In: Vol 1. Explorations in quality assessment and monitoring. Ann Arbor, Michigan: Health Administration Press, 1980.

60.

Al-Shaqsi

SZK

. Response time as a sole performance indicator in EMS: Pitfalls and solutions. Open Access Emergency Medicine 2010; 2: 1–6.

61.

The Australian Health Practitioner Regulation Agency. Paramedicine Board of Australia - Code of conduct (interim). Paramedicine Board of Australia. 2018.

62.

National Quality Board. Quality in the new health system-Maintaining and improving quality from April 2013. London; 2013.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.50 MB

0.02 MB

0.00 MB