Sage Journals: Discover world-class research

Abstract

The importance and value of engaging patients and the public as co-researchers (i.e., “patient engagement in research”) is becoming more evident, and guiding methods must be available for researchers conducting their work at different points along the engagement spectrum. This article provides a virtual workshop model for integrating patient and public stakeholder perspectives in data analysis and interpretation. The model is based upon a critical reflection on the methods that underlaid the consultation stage of our scoping review on patient and caregiver preferences for cardiac surgery. It involves four virtual workshop sessions held on separate days, each achieving the unique goals of (a) establishing participants’ technological literacy within the virtual platform, (b) obtaining responses to the research question, (c) introducing participant perspectives into research analysis and interpretation, and (d) prioritizing research findings or future research agendas. Further, a description of the considerations related to virtual engagement, including those pertaining to equity, diversity, and inclusion; features of the virtual platform; and roles for the research team are provided. This paper contributes toward a methodological toolkit for patient engagement in research, especially as an adjunct to research with otherwise minimal patient engagement. It also adds to the emerging literature on practical approaches to patient engagement in research as more engagement is occurring virtually.

Keywords

patient-oriented research patient and public involvement stakeholder consultation

Introduction

Patient engagement in research is an approach that maintains that the lived experience of health system users (i.e., patients and caregivers) contributes unique insights to research aimed at studying their health condition and/or improving their care (Supplemental Table 1). As opposed to research where health system users are passive participants, patient engagement actively involves members of the public as “co-researchers” or “patient partners.” In this way, research is not done for or about patients but rather with and by them (National Institute for Health Research, 2012). The importance of involving patients and the public in research and decisions impacting their care is becoming increasingly apparent. It has been postulated that as the funders of a public healthcare system and participants of research, patients and the public have a fundamental right to be actively involved in the design and conduct of health research (Duffett, 2017). There are also benefits within public and privatized healthcare systems and industry in terms of the uptake of research findings that incorporate the preferences of its primary stakeholders (Vat et al., 2020). Studies have even demonstrated measurable improvements in several drivers of clinical outcomes as a result of patient engagement (e.g., health resource utilization, participation in risk reduction, improved scores on psychometric testing) (Edgman-Levitan, 2013; Laurance et al., 2014; Osborn & Squires, 2012; National Voices, 2015).

Despite the fact that patient engagement in research is more frequently a requirement for grant applications, trial reporting, and in the journal submission process, the relationship between patients and researchers is still largely transactional (Chalmers & Glasziou, 2009; Liberati, 2011; Thornton, 2014; Tinetti & Basch, 2013). Patients and the public are primarily engaged during research design and priority setting and much less often in analyzing and evaluating research results (Domecq et al., 2014). Other non-methodological barriers to patient engagement in research include the traditional involvement of highly objective, quantitative methods in clinical studies and that the culture of biomedicine as an institution has not fostered patient engagement as a high priority (Boutin et al., 2017; Gagliardi et al., 2008). It is becoming increasingly evident, however, that engagement exists on a spectrum (International Association for Public Participation, 2018; Manafò et al., 2018) that can be tailored to the resources available at the researcher’s discretion as well as the experience and interests of patients and the public and other members of the research team.

Though there is an abundance of literature available on its conceptual aspects, few pragmatic methodologies (i.e., “how to’s”) are explicitly designed for patient engagement in research (Kirwan et al., 2017; Smith-MacDonald et al., 2019). Still, the most common approaches to patient engagement involve qualitative techniques (e.g., focus groups, qualitative surveys) (Domecq et al., 2014; Phoenix et al., 2018). To maximize the benefits of engagement and make the process more accessible for non-experts, it is important to provide opportunities for patient engagement across different stages of the research cycle, and to develop methodological models and frameworks which address the limitations of time, financial, and personnel resources, and those inherent to the nature of clinical research (i.e., an emphasis on objectivity, measurable evidence, etc.). To facilitate the accessibility of engagement, guiding methods must be available for researchers, patients, and the public at different points along this spectrum and with varying degrees of familiarity with patient engagement in research, and qualitative research designs.

Given the lack of pragmatic methodologies for patient engagement in research at different stages of the research process, the purpose of this article is to present a model for virtual consultation of patients and the public during data analysis and/or interpretation. The context for model development was a scoping review study aimed at understanding the care preferences and prioritized outcomes of cardiac surgery patients and caregivers (Oravec et al., 2021a, 2021b). While patients were part of our team from conception to translation, scoping review methodologies traditionally involve a formal, post hoc consultation stage with research stakeholders. It was during this stage that we conceptualized a model for consulting patients and the public during the data analysis and interpretation phases of research, which employs a deductive approach informed by focus group discussions to data that have already undergone preliminary analyses. This article and the model described herein are intended to serve as a template for researchers who seek to incorporate patient engagement in studies with otherwise limited engagement of health care stakeholders. The model also addresses researcher bias—and thereby strengthens the trustworthiness and rigor of research findings—by minimizing participant exposure to predetermined ideas about the interpretation of research findings. This is especially important in qualitative research, because its focus is on the subjectivity of people’s experiences (Smythe & Giddings, 2007).

Method

Contextualizing Model Development: Introducing Our Scoping Review Case Example

The model is informed by our cumulative expertise and experience with patient engagement in research, and was developed through a critical reflection of our prior research, a scoping review. The review identified patient and caregiver preferences and prioritized outcomes related to cardiac surgery. Details of the study protocol (Oravec et al., 2021a) and findings from the review (Oravec et al., 2021b) are found elsewhere. In brief, the scoping review involved a qualitative coding process and generated a thematic summary of patient and caregiver perspectives available from the literature (Ibid). Because these perspectives were based on indirect responses to the research question, we aimed to validate the findings by posing the question directly to cardiac surgery patients and caregivers through a formal “consultation workshop.” The model’s description draws from our experience planning and conducting this workshop, which comprised four sessions and primarily gathered participant insights through focus groups.

The model was designed as a means to introduce patient and public value perspectives in research analysis and interpretation of primary or secondary qualitative or quantitative data. It assumes that researchers have already gathered data and conducted a preliminary analysis, and are now seeking to validate its findings and investigate their applicability using a stakeholder engagement approach.

Our Research Team

Our research team encompasses a diverse range of academic and experiential backgrounds, including healthcare providers, and experts in patient engagement, qualitative and clinical research, and library science. The workshop’s focus groups were co-facilitated by research team members: NO, AG, BB, CM, MD, AS, and AC, with additional contributions from RA and TD. Workshop participants were aware of researcher titles and affiliations, relevant experiences and training, and motivations for participation in the research project. Only BB had prior relationships with workshop participants, as two individuals were recruited through his social networks.

Patient and Public Involvement

This report follows the Guidelines for Reporting Involvement of Patients and the Public-2 (Supplemental Table 2) (Staniszewska et al., 2017). The patient co-researchers on our team (AG, BB) collaborated on the conceptualization and design of the consultation workshop and were also involved in its conduct in the roles of presenter, facilitator, and reporter (vide infra). These individuals’ involvement in the consultation workshop helped to bridge the divide between non-cardiac surgery patient members of the research team and the workshop participants and helped ensure that patient and public perspectives were driving our research. It also served to reinforce our belief that lived experience is a valued and equal form of expertise and that our workshop was designed with this patient-centered approach in mind. Both BB and AG meet the International Committee of Medical Journal Editors criteria for authorship, and we acknowledge their invaluable insights in the development of this model (Richards et al., 2020; Ellis et al., 2021).

Recruitment into the Virtual Consultation Workshop

Participants were recruited through a combination of professional and social networks, public advertisement, and convenience sampling from an institutional database of cardiac surgery patients who had previously consented to be contacted for research. Since the database included only patients' contact information, snowball sampling was employed to recruit the informal caregivers of contacted patients. In addition to being an adult cardiac surgery patient or caregiver at our study hospital (St. Boniface Hospital, Winnipeg Canada) at least 6 months prior to the first workshop focus group, eligibility criteria included: (a) access to the internet and an electronic device, (b) a willingness to reflect on personal experiences relating to surgery and share these in a group setting, (c) availability during the scheduled meeting times, and (d) the ability to speak and read English. In total, two individuals were recruited through professional and social networks, and of the 27 individuals contacted through our research database, 16 agreed to participate. Prior to the first workshop session, one individual dropped out due to health-related reasons that prevented participation. Consequently, 17 individuals (10 patients and 7 caregivers) participated in the workshop. Each participant received a $10 honorarium for attending the first technology-focused workshop session (held November 24, 2021) and a $25 honorarium per each of the other workshop sessions (held November 27–29, 2020). Institutional review board approval was obtained in May 2020 (HS23739). All participants provided written informed consent prior to workshop participation.

Model for Integrating Patient and Public Perspectives in Research Analysis and Interpretation

Figure 1 displays the developed model for integrating patient and public perspectives in research analysis and interpretation. The model illustrates each workshop session’s key components and considerations and suggests that workshop sessions build upon and inform each other. For example, the information gathered from participants in Session 2 is applied to the revision of a preliminary analysis or conceptual summary of research that has already been conducted. The appropriateness (i.e., patient-centeredness, comprehensibility, etc.) of the revised analysis/summary is adjudicated by participants in Session 3. Session 4 is an optional priority-setting activity for the research findings or future research. We further expand upon the details of the model below.

Figure 1.

Overview of workshop model for introducing patient and public perspectives in research analysis and interpretation.

Session 1: Familiarization with the Virtual Platform

Key Considerations

Ahead of the main workshop sessions, it is important to provide participants with the opportunity to familiarize themselves with the virtual platform and learn about and perform the technical skills required to actively participate in the workshop. We suggest this occur through a written document that outlines the technology-related details and competencies required to engage in the sessions, as well as an optional initial session that allows participants to demonstrate these skills ahead of the main workshop sessions.

Application in Case Example

Ahead of Session 1, workshop participants were provided with written and illustrated instructions on how to download and install the virtual platform (i.e., Zoom) to their device and how to access the platform directly through a Web site, follow a link to join a session, change their display name and/or picture (to match personal privacy-related preferences), and perform the core competencies required to participate in the workshop. These instructions were tailored to the major types of devices participants could access the sessions through (i.e., computers, Android tablets or phones, iPads or iPhones). The core competencies described included how to: (a) mute and un-mute microphones; (b) turn video cameras on and off; (c) raise a “virtual hand” (for example, to ask a question); (d) and use the chat, whiteboard, and screen sharing functions. These instructions also provided an overview of the free, online survey tool, “Survey Monkey,” used to collect feedback after each session and for a ranking exercise that took place in Session 3. At the end of the instructions, participants were provided with a link to a practice survey that had the dual purpose of exposing participants to the different question types (i.e., Likert-like scale, short answer) they would encounter in session surveys and asking participants about their comfort with and ways in which we could better support their participation in the workshop. Participants were also provided with the contact information of an individual on the research team who could provide one-on-one support.

During the session itself, participants were assigned to breakout groups facilitated by members of the research team (n = 5–6 participants per group). Facilitators provided overviews of each core competency, after which participants were asked to perform the competency themselves. The session closed with the participants being asked to provide feedback on the session through an anonymous survey administered through Survey Monkey.

Overview of the Structure of Sessions 2–4

The main workshop comprised three sessions, held on separate consecutive days, each lasting between one-and-a-half to 2 hours. In contrast to a single, full-day workshop, this sequence provided more time for pre-and post-session reflection and, in our experience, a richer exchange of ideas. It also facilitated idea generation by allowing for pre-session (mandatory and/or optional) readings and post-session “homework.” The readings include the upcoming session’s agenda and provide a summary of the information to be discussed in order to accommodate different learning styles and preparation preferences. “Homework” questions consist of the focus group discussion prompts. Sending the prompts out ahead of each session allows participants time to generate ideas and prepare to share their experiences. These files are available through the corresponding author or their lab’s Web site (www.patientengagementinresearch.ca).

Session 2-4 should begin with a welcome and meeting overview, and an overview of guidelines for virtual interactions (including the opportunity to append the list). Similar to recommendations for researcher and participant conduct in focus group research (Krueger & Casey, 2015), guidelines for virtual interaction help to establish an environment of mutual respect, civility, and inclusion, but with special considerations made for the virtual format. An example of guidelines for virtual interaction, based upon the case example, is displayed in Supplemental Table 3. The general structure of each of these sessions also includes: (a) a presentation of background information (that overlaps with and complements the pre-reading material), (b) breakout focus group-style discussions, (c) reporting back from the breakout groups, and (d) closing remarks and post-session evaluation. Given the group size and time limits we chose to incorporate icebreaker activities into Session 1 and Session 2’s breakout group. All three main sessions have the same key features (Table 1), but each have unique goals (Table 2) which together accomplish the overall workshop aim, which is to integrate patient and public stakeholder perspectives in research analysis and interpretation.

Table 1.

Key Common Features of the Main Sessions (2–4).

Feature	Description	Why It is Important
Homework questions (pre-session)	Focus group discussion prompts sent out ahead of time (i.e., day before the associated session)	So that participants can begin to generate ideas and prepare to share their experiences
Welcome/meeting overview	Research team and participant introductions, presentation of guidelines for virtual interaction, sharing of session goals	Help to establish environment of mutual respect, civility, and inclusion
Presentation of background information	Relevant information needed to participate (e.g., research or methods background, etc.)	Overlaps with and complements pre-reading (homework) material
Breakout small-group discussions	Focus groups	Integrates patients and public in the research process, generates data to validate and modify preliminary analyses
Reporting back from small groups	Participants from each breakout focus group summarize group discussion	Integrates patients and public in the research process, reinforces participant conceptions of the most important messages from the focus group discussions
Closing remarks and post-session evaluation	Overview of upcoming session’s pre-reading and homework and opportunity to provide anonymous feedback on the session	Preparation for next session, feedback for improvement

Table 2.

Content and Expected Outcomes of the Main Sessions (2–4).

Agenda Items	Expected Outcomes
Session 2
Content area background	Familiarization with the content area
Workshop goals	Increased understanding of research objectives
Key definitions	Clarification of high-level concepts
Breakout room focus groups	Direct (unbiased) responses to the research question
Large group summary	Idea sharing and preliminary theme building
Session 3
Research methods presentation	Familiarization with the methods used to obtain indirect responses to the research question (e.g. research synthesis, research not involving patients and public, etc.)
Knowledge summary	Presentation of the revised qualitative research summary (i.e. based on indirect responses to the research question and the results of session one)
Breakout room focus groups	Feedback on the (a) qualitative research summary and (b) biased responses to the research question
Large group summary	Idea sharing and continued theme building
Session 4 (optional)
Explanation of the prioritization activity	Familiarization with the prioritization activity
Prioritization activity	Creation of a patient and public-prioritized research or implementation agenda

Session 2

Key Considerations

Session 2 serves as the participants’ formal introduction to the research project. It should include a clarification of workshop goals and a presentation of the content area background. To minimize discrepancies introduced by participant misunderstanding, it is also important to clearly define higher-level concepts at this time. Following this initial presentation, participants can be divided into breakout rooms that follow considerations related to optimal focus group size (Krueger & Casey, 2015). In these breakout groups, the research question is re-phrased to address participants directly. The data generated at this stage represent unbiased responses to the research question since participants have not been exposed to the researchers’ preliminary analysis results. Following the focus group discussions, the breakout rooms are dissolved, and a designated representative from each group provides a summary of their group’s responses. During this activity, both the participants and researchers may begin to notice commonalities (preliminary themes) in the group responses.

Post-Session Analyses

After Session 2, the task of the researchers is to analyze the focus group responses and modify their thematic analysis with this new information. A variety of methods have been described for qualitative data processing (Castleberry & Nolen, 2018). This format (uniquely) provides a way to deductively validate the researcher’s inductive analysis. That is, the researcher’s preliminary framework—developed from the “ground up” using data from the research project—is tested by how well new participant responses can be categorized within the existing framework. One of three scenarios can occur: (a) participant responses validate the existing thematic construction, (b) participant responses change the interpretation of the other research such that the thematic constructions should be modified, or (c) participant responses contradict the interpretation of the other research. Regardless of which of these scenarios occurs, the outcome of Session 2 is a revised thematic summary that reflects patient and public perspectives as opposed to one that is solely based on the perspectives of the research team.

Application in Case Example

In our case example, Session 2 began with a presentation that familiarized participants with the concept of Enhanced Recovery After Cardiac Surgery (ERAS® Cardiac) – a list of 22 recommendations for faster and more complete medical and functional recovery developed by clinician content experts (Engelman et al., 2019). We also shared our goals for the workshop, which were to: (a) obtain patient and caregiver perspectives on recovery after cardiac surgery (Session 1), (b) share and validate the findings of a scoping review on the same topic (Session 2), and (c) rank a revised list of patient and caregiver-derived preferences and prioritized outcomes (Session 3). Participants were then placed into one of four breakout rooms that consisted of two rooms with cardiac surgery patients, one with caregivers, and one with patient-caregiver “dyads” (i.e., patients and caregivers who attended the meeting using the same electronic device). The focus group discussion prompts were the scoping review research questions rephrased to address patients/caregivers directly. For instance, the scoping review’s primary research question was, “What does the existing literature say about patient and caregiver-identified preferences and outcomes as they relate to care received in the perioperative period of cardiac surgery and the lifelong impact of cardiac surgery on the patient?” When posed to workshop participants, the question became, “What were your preferences for the perioperative period of cardiac surgery (before, during, after, and long after)?”, “What outcomes were important to you for cardiac surgery?”, “How would you know the surgery was successful for you?”, and “What did you hope life would be like after surgery?” Though these discussion prompts were not pilot tested, they were developed in collaboration with patient partners, who critiqued the applicability of the questions from the perspective of individuals with lived experience of undergoing cardiac surgery.

Following the session, the research team conducted a thematic analysis of participant responses which validated the themes generated in the preliminary analysis. This analysis also provided novel data that addressed the scoping review’s underlying research question. Two coders (NO and AMC) reviewed meeting notes and recordings in duplicate, and assigned participant statements into existing or novel themes. The same researchers were involved in the coding process that produced the preliminary framework (i.e., the scoping review), so they had familiarity with the nature of responses to the research questions. The thematic coding process involved multiple iterations. Consensus was achieved through discussion between the two coders, and with the rest of the research team in cases of conflict.

Session 3

Key Considerations

It is important to begin Session 3 by engaging participants in effective knowledge translation. This involves familiarizing participants with the methods used to obtain indirect responses to the research question and how the data they generated in Session 2 mapped onto the researchers’ original preliminary analyses. Whereas the goal of Session 2 is to obtain unbiased participant responses to the research question, the goal of Session 3 is for participants to assess the validity of the researcher-derived thematic constructions. In this way, the workshop focus groups serve the dual purpose of soliciting additional data (Session 2) and introducing patient and public perspectives into the original preliminary analyses (Session 3). During this session’s focus group, participants may offer additional responses to the research question(s) posed during Session 2. This may occur because examples of the types of responses found in the literature may prompt new ideas. Like the previous session, Session 3 adjourns with a summary of the individual breakout room discussions. This format helps to integrate stakeholders in the research process and reinforces participant conceptions of the most important messages from the breakout room discussions.

Post-Session Analyses

The task of the researchers following Session 3 is similar to Session 2. In preparation for prioritizing research findings or future research agendas in Session 4, the findings from the focus group discussions should be applied to the revised thematic summary in another iteration. The revised thematic should then be transformed into a document that will support Session 4’s activities.

Application in Case Example

In our case example, Session 3 began by providing background on scoping reviews, specifically their goals and underlying methods, and a description of the strategy used to map the literature on patient and caregiver preferences and prioritized outcomes for cardiac surgery. Following this presentation, the research team presented the revised thematic summary based on the original preliminary analyses (i.e., the scoping review) and the results of Session 2. We deliberately shared which themes were modified or added based on Session 2. This was intended to summarize and reinforce Session 2’s findings and ensure that participant responses were correctly interpreted. The content of Session 3’s focus groups was meant to be very open-ended, as reflected in the discussion questions: “Do the findings of the review ‘resonate’ with your experiences?” and “What (if anything) is missing from the findings, as informed by your own experiences and the ideas generated at yesterday’s meeting?” After Session 3, the session’s responses were analyzed in anticipation of Session 4’s activities.

Session 4

Key Considerations

Session 4 involves an optional prioritization activity that can be used to rank or inform the implementation of research findings or set an agenda for future studies. This format has been described in other frameworks for patient and public engagement (Khodyakov et al., 2020). Detailed descriptions of group consensus techniques, such as the Delphi process or nominal group technique, are readily available in the literature (Gallagher et al., 1993; Nelms & Porter, 1985; Rowe & Wright, 2001). The research team must achieve a number of prerequisites prior to engaging participants in Session 4. Namely, the researchers must be able to produce a thematic summary that is reflective of participants’ Sessions 2 and 3 responses (i.e., to the research question(s) and regarding the preliminary thematic framework, respectively). A potential constraint may be time, especially if the sessions are scheduled in close proximity to each other. Another prerequisite is the availability of a survey or items that summarize the thematic framework in a series of questions reflective of the session’s specific goals. We expand on this concept in the proceeding case example. We have found that advanced preparation for the session can facilitate a timely yet comprehensive analysis and creation of relevant prioritization material. Alternatively, Session 4 may be held several days or weeks after Sessions 2 and 3 without significant limitations.

Application in Case Example

Ahead of session 4, themes of individual preferences and/or outcomes into statements which the participants ranked according to their agreement on a Likert-style scale from 0 to 10. For example, the preference sub-category, “Family,” within the theme, “Social support” was converted to the statement, “Family/friend support is important to patients. Examples include having visitors in-hospital, staying in-touch, and/or family and friends that are involved in their care and recovery.” Session 4 opened with a presentation that described our chosen group consensus technique (i.e., modified Delphi process). Participants were then divided into breakout groups according to their primary stakeholder identity (i.e., patient or caregiver). Patient-caregiver dyads attending the session through shared devices were allowed to join either group. Three of the four dyads chose to join the caregiver group. In the breakout rooms, participants were provided with a survey link specific to each stakeholder group’s survey, hosted on Survey Monkey. Once a given breakout room’s participants had completed the survey, the researchers analyzed the results in real time. A summary was generated listing the median ratings, first and third percentiles, and interquartile ranges for each statement. The proceeding discussion then focused on the statements with the highest interquartile range, continuing to statements with greater agreement until the group felt ready to re-rank the items. At this point, the survey was administered a second time, and the meeting adjourned. Participants were provided with a summary of the workshop findings once all analyses were complete.

Other considerations

Equity, Diversity, and inclusion

Equity, diversity, and inclusion (EDI) considerations are of increasing importance within research and are essential to workshop design and conduct. These could include the thoughtful integration of a diversity of perspectives and identification and mitigation of barriers (e.g., environmental, systemic) to workshop engagement. For example, women are under-represented in cardiac surgery research, and the disease course is different between men and women. Therefore, when forming our research team, we sought patient partners that represented the perspectives of both men and women. We also aimed to have diverse representation among workshop participants using the personal information available to us in the recruitment database. We actively worked with workshop attendees to identify any accessibility barriers, including a formal technology and other accommodation needs survey and the development of Session 1 (which helped ensure any outstanding barriers to participation were identified and addressed). The integration of EDI considerations led to a broader range of perspectives being reflected in our results, including a model that is hopefully more relevant and accessible to a wide range of individuals.

The Virtual Platform

We chose to use the proprietary video conferencing platform, “Zoom” to host the virtual workshop, partly due to its widespread uptake as a method of virtual communication during the COVID-19 pandemic. A license was purchased for the “Zoom for Healthcare” plan, which supports the security and privacy standards outlined by the Health Insurance Portability and Accountability Act, the Personal Information Protection and Electronic Documents Act, and the Personal Health Information Protection Act (Zoom Video Communications, 2021). In addition to its compliance with these regulations, the platform offers a number of features which facilitate its use for focus group discussions and meeting our model’s aims. For example, researchers have the option of recording meetings, sharing their screens, integrating real-time polls, and sorting attendees into small-group “breakout rooms.”

Research Team Member Role Descriptions

As displayed in Table 3, we have outlined six basic roles for research team members which can easily be modified, collapsed, or expanded depending on the individual study needs. Our research team members shared and alternated between multiple roles throughout the workshop. Many of these roles are regularly suggested for focus group research.

Table 3.

Suggested Research Team Member Roles and Responsibilities During the Workshop.

Role	Role Responsibilities	Considerations
Meeting leader	Ensures session objectives met, timekeeper, point of contact for participants and research team members	Beneficial to appoint a leader with experience conducting large meetings and with patient engagement (Deakin & Wakefield, 2014; Forrestal et al., 2015)
Technology administrator	Resolving technology issues, answering technology-related questions, creating a technology information handout	Allocating these responsibilities to a single member of the research team allows other members to focus on their own tasks and prevented disruption (Kite and Phongsavan, 2017)
Presenter	Knowledge translation (e.g., content area background, research methods presentation, etc.)	This role can be shared over the course of the workshop to showcase the expertise of different members of the research team
Facilitator	Guides group discussions to meet research aims; fosters comfortable environment for participation	Given the complexity of managing group dynamics and while simultaneously processing and reflecting the dynamic content of group-based conversations, it is important to have skilled facilitators leading the focus groups (Robinson, 1999; Nyumba et al., 2018)
Note taker (transcriber)	Transcribes important messages from focus group discussions	It is important to have the notes be visible to participants as they are being taken to ensure accuracy of what is being captured as well as shared understanding of what is being said
Reporter	Shares summary of focus group discussions with large group	This role can be offered to participants to support a greater sense of shared ownership of the findings and ensure that the key messages are grounded in the interpretations of the research stakeholders

Participant Feedback

In consideration of the limited evidence base for virtual focus groups (Morgan, 2019), others have stressed the importance of an evaluative component as part of the study design (Daniels et al., 2019). Thus, following each session, workshop participants were asked to complete a survey that assessed their experience of the session. The two survey questions were, “What did you like about the meeting?” and “Do you have any suggestions about how the meeting could be improved?” Participant responses to the surveys were between 54 and 85%. The comments for each question were analyzed thematically and are presented in Table 4.

Table 4.

Thematic Summary of Participant Workshop Experiences.

Theme	Examples
Technology session
Confidence in Zoom	“This was very informative…nice to know how the controls work… will have confidence moving forward”
Confidence in Zoom	“Good to become familiar with the Zoom app options”
Overcoming technical issues	“There was lots of help to overcome glitches”
Overcoming technical issues	“Saw that there were a number of technical issues, but we managed to work them out”
Research team	“Everyone was patient”
Research team	“Very helpful researchers.”
Format	“We liked that it was quite casual.”
Sessions 1–3
Organization	“It was well organized and moved quickly.”
	“Good leadership.”
	“It was well organized and expectations were very clear.”
	“The meeting format is clear and organized.”
Preparation	“Everything was well explained re: meeting format, goals, etc.”
Participation	“Gave everyone an opportunity to participate.”
Respect	“Members were all very respectful.”
Meeting peers	“I liked the opportunity to hear the other caregiver experiences, suggestions, etc. to know that they had similar thoughts and issues.”
	“Found out there are other persons with same stress levels, same questions, and worries as myself when it came to the heart surgery.”
	“Learned a lot of other participants’/caregivers’ experiences. Reassuring to hear these experiences – supportive.”
Group size	“It was easy to share info with the much smaller group.”
Duration	“[Session 2] felt a bit rushed, especially since we didn’t know how much time each speaker would have until moving on to wrapping up the thoughts and ideas.”
	“There was more time to share today [Session 3] which was very important.”
	“Had lots of time for breakout groups [Session 3].”
Content	“More focus to the point, there was some introduction [Session 2] that didn’t seem necessary and was explained a few times that could have been spent on the content discussion.”
Content	“It was clear and concise [Session 3].”
Virtual platform	“The Zoom call went smoothly from my end. The picture and sound quality was good. This is the first time we have used Zoom.”
Research team	“The quality of listening skills presented by the team is a credit to their dedication to this project.”

Discussion

We have presented a virtual workshop model for integrating patient and public perspectives in research analysis and interpretation. Our work adds to an emerging knowledge base of methods for patient engagement at a time when there has been an increased interest in virtual health research. The model is particularly applicable when longitudinal engagement is not feasible, and for studies utilizing secondary data. It can also be used to introduce a greater range of patient and public perspectives within studies that already engage a small number of patient and public co-researchers.

The COVID-19 pandemic accelerated many changes in research conduct and led to a growing body of literature on the virtual conduct of qualitative research (Archibald et al., 2019; Dodds & Hess, 2021; Lobe et al., 2020; Teti et al., 2020). As others have identified, virtual platforms can both facilitate and hinder accessibility depending on a study’s target population (O ’Connor et al., 2011; Sy et al., 2020). Our model addresses accessibility barriers through participant handouts and a session dedicated to establishing the core competencies required for virtual workshop engagement. In this way, we are among the first to report detailed recommendations for ensuring technological literacy among virtual focus group participants. In addition, our explicit description of technological competencies, features of the virtual platform, and timepoints for virtual contact make our model more accessible for researchers who are have limited familiarity with conducting their work online. While some scholars have suggested that virtual methods for participatory research are not a substitute for in-person methods (Goldstein et al., 2020; Teti et al., 2021), this notion may be explained by a lack of familiarity and evidence for the conduct of qualitative research online, as opposed to the approach being inherently inferior. More research is needed on the patient and public experience of virtual engagement.

Other methods have been described for asynchronous forum-based focus groups and more general considerations for virtual qualitative research (Hallam, 2021; Roberts et al., 2021). The model reported here is also not the first to offer a multi-session approach to stakeholder involvement in research analysis, nor is it the first to involve stakeholders longitudinally throughout a literature review (Stocker et al., 2021; McCarron et al., 2021; Pham et al., 2014). Our approach is unique, however, in its emphasis on formal focus group methodology and the combination of inductive and deductive approaches to summarizing indirect responses to a research question, and validating them by directly addressing the primary stakeholder. In this way, we address a challenge of research synthesis: that high-level questions may not be addressed directly in the literature or through qualitative summaries of the results of quantitative research designs. For instance, in our case example, the concept of patient and caregiver “preferences” proved to be ambiguous. A number of articles reported “satisfaction” as a patient-reported outcome, which did not ultimately address patients’ own preferences but rather the degree to which they tolerated a particular intervention or care paradigm. Other times, studies reported negative constructions of preference (“do not want/like/need”) which were similarly vague in identifying what patients preferred instead. The consequence of this ambiguity is that there is a potential for incongruence between responses to the research question identified by the review and when it is posed directly to stakeholders. Thus, in our model, the second workshop session serves to validate the results of the thematic analysis through a deductive approach. Whereas the researchers construct the thematic analysis (or other qualitative analytical models) inductively by summarizing individual responses to the research question, the validity of the thematic construction is tested by the output of the focus group discussions when the research question is posed directly to workshop participants.

Table 5 summarizes the accomplishments of our model and outlines areas of ongoing development. Some important challenges and limitations should be discussed and addressed by future research. One is the requirement for significant time resources, particularly in the planning stages of the workshop. This may have implications on methodologic rigor. For instance, in our case example, the modified Delphi process did not adhere to a specific threshold for inclusion as there was only enough time for a single round of re-ranking. Second, the eligibility criteria of the case study used to inform model development may have posed participation barriers to those experiencing social vulnerability. These included criteria requiring access to the internet and an electronic device, and availability to attend the scheduled workshop times. Thus, the model may not include considerations relevant to the needs of these populations. To help increase participant diversity, researchers seeking to apply this model should also consider targeted recruitment strategies (Brouwer & Hessels, 2019) and improving the accessibility of the virtual workshop sessions to individuals of all socioeconomic statuses through strategies like identifying access points (e.g., community access hubs or libraries with free loaner devices, internet access, and available private rooms) for individuals without access to the internet and/or electronic devices, planning workshop sessions around participants’ schedules, and covering other costs related to workshop participation (e.g., cost of hiring a babysitter or other caregiver to support workshop attendance, transport to technology access point).

Table 5.

Summary of Model Accomplishments and Areas for Ongoing Development.

Accomplishments

Involving patients and the public in analysis and interpretation phases of research

Minimizing researcher bias in the interpretation of research findings

Integration into the standard research workflow

Areas for future consideration

Ways to increase accessibility (e.g., barriers related to virtual attendance)

Ideal session duration and time needed for between-session analyses

Exploring other qualitative methods in analysis of participant responses

Ongoing refinements to the model based on collective researcher and patient and public experiences

Finally, it is important to consider how the subjectivities of the researchers might have influenced the study. Several researchers occupy roles in clinical settings, and this may have influenced perceptions of power dynamics, and the interpretation of participant responses to the research questions as to overemphasize medical aspects of recovery after surgery. Patient partners on the research team may have prioritized participant responses that resonated with their own experiences after surgery. Conversely, those without experience as either care providers or patients may have misinterpreted or dismissed certain responses due to lack of familiarity.

Conclusions

Our virtual workshop model makes patient and public consultation more accessible during a critical stage of the research cycle, and one that less often involves patient engagement. The model employs a deductive approach to validation of an inductively-developed qualitative research summary, which is well-suited for its application at the analysis and interpretation stages of research. It contributes towards a methodological toolkit for patient engagement in research, especially as an adjunct to research with otherwise minimal patient engagement. It is our hope that this article stimulates discussion and ongoing developments in patient engagement methodology.

Supplemental Material

Supplemental Material - A Virtual, Multi-Session Workshop Model for Integrating Patient and Public Perspectives in Research Analysis and Interpretation

Supplemental Material for A Virtual, Multi-Session Workshop Model for Integrating Patient and Public Perspectives in Research Analysis and Interpretation by Nebojša Oravec, BSc, Annette S.H. Schultz, RN, PhD, Brian Bjorklund, BA, April Gregora, BA, Caroline Monnin, BA, MLIS, Mudra G. Dave, BKin, Todd A. Duhamel, PhD, Rakesh C. Arora, MD, PhD, and Anna M. Chudyk in International Journal of Qualitative Methods

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research,authorship,and/or publication of this article: RCA has received an unrestricted educational grant from AVIR Pharma Inc,and honoraria from Abbott Nutrition and Edwards LifeSciences for work unrelated to this manuscript. The other authors declare that they have no competing interests.

Funding

The author(s) disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: This study’s patient and public involvement activities were supported by a George & Fay Yee Centre for Healthcare Innovation’s Preparing for Research by Engaging Patient and Public Partners (PREPPP) Award. AMC’s postdoctoral fellowship is funded by a Canadian Institutes of Health Research Patient-Oriented Research Awards - Transition to Leadership Stream fellowship. NO was supported by the American Association of Thoracic Surgery (AATS) Foundation Summer Intern Scholarship in Cardiothoracic Surgery and a stipend from the University of Manitoba Medicine Summer Research Program. Funders had no role in any aspect of the scoping review or consultation workshop.

Research Ethics and Patient consent

Institutional review board approval was obtained in May 2020 (HS23739). All participants provided informed consent prior to participation in the consultation workshop.

ORCID iD

Nebojsa Oravec

Supplemental Material

Supplemental material for this article is available online.

References

Archibald

M.M.

Ambagtsheer

R.C.

Casey

M.G.

Lawless

(2019). Using Zoom videoconferencing for qualitative data collection: Perceptions and experiences of researchers and participants. International Journal of Qualitative Methods, 18, 1–8. https://doi.org/10.1177/1609406919874596

Boutin

Dewulf

Hoos

Geissler

Todaro

Schneider

R. F.

Sargeant

(2017). Culture and process change as a priority for patient engagement in medicines development. Therapeutic Innovation and Regulatory Science, 51(1), 29–38. https://doi.org/10.1177/2168479016659104

Brouwer

Hessels

L. K.

(2019). Increasing research impact with citizen science: The influence of recruitment strategies on sample diversity. Public Understanding of Science, 28(5), 606-621. https://doi.org/10.1177/0963662519840934.

Castleberry

Nolen

(2018). Thematic analysis of qualitative research data: Is it as easy as it sounds? Currents in Pharmacy Teaching and Learning, 10(6), 807–815. https://doi.org/10.1016/j.cptl.2018.03.019

Chalmers

Glasziou

(2009). Avoidable waste in the production and reporting of research evidence. Lancet, 374(9683), 86–89. https://doi.org/10.1016/S0140-6736(09)60329-9

Daniels

Gillen

Casson

Wilson

(2019). STEER: Factors to consider when designing online focus groups using audiovisual technology in health research. International Journal of Qualitative Methods, 18(1), 1-11. https://doi.org/10.1177/1609406919885786.

Deakin

Wakefield

(2014). Skype interviewing: Reflections of two PhD researchers. Qual Res. https://doi.org/10.1177/1468794113488126

Dodds

Hess

A. C.

(2021). Adapting research methodology during COVID-19: Lessons for transformative service research. Journal of Service Management, 32(2), 203–217. https://doi.org/10.1108/JOSM-05-2020-0153

Domecq

J. P.

Prutsky

Elraiyah

Wang

Nabhan

Shippee

Murad

M. H.

(2014). Patient engagement in research: A systematic review. BMC Health Services Research, 14, 1–9. https://doi.org/10.1186/1472-6963-14-89

10.

Duffett

(2017). Patient engagement: What partnering with patient in research is all about. Thrombosis Research, 150, 113–120. https://doi.org/10.1016/j.thromres.2016.10.029

11.

Edgman-Levitan

(2013). Partnering with patients, families, and communities for health: A global imperative. WISH patient and family engagement report 2013. https://www.wish.org.qa/wp-content/uploads/2018/01/27425_WISH_Patient-Engagement_web-1.pdf%0Ahttps://workspace.imperial.ac.uk/global-health-innovation/Public/WISH_reports/27425_WISH_Patient_Engagement_web.pdf, pp. 5–26.

12.

Ellis

Kitchin

Vis-Dunbar

(2021). Identification and reporting of patient and public partner authorship on knowledge syntheses: Rapid review. Journal of Participatory Medicine, 13(2), Article e27141. https://doi.org/10.2196/27141

13.

Engelman

D. T.

Ben Ali

Williams

J. B.

Perrault

L. P.

Reddy

V. S.

Arora

R. C.

Boyle

E. M.

(2019). Guidelines for perioperative care in cardiac surgery: Enhanced recovery after surgery society recommendations. JAMA Surgery, 154(8), 755–766. https://doi.org/10.1001/jamasurg.2019.1153

14.

Forrestal

S. G.

Angelo

A. V. D.

Vogel

L. K.

(2015). Considerations for and lessons learned from online , synchronous focus groups. https://doi.org/10.29115/SP-2015-0015

15.

Gagliardi

A. R.

Lemieux-Charles

Brown

A. D.

Sullivan

Goel

(2008). Barriers to patient involvement in health service planning and evaluation: An exploratory study. Patient Education and Counseling, 70(2), 234–241. https://doi.org/10.1016/j.pec.2007.09.009

16.

Gallagher

Hares

Spencer

Bradshaw

Webb

(1993). The nominal group technique: A research tool for general practice? Family Practice, 10(1), 76–81. https://doi.org/10.1093/fampra/10.1.76

17.

Goldstein

R. Z.

Vasques

R. A.

Loschiavo dos Santos

M. C.

(2020). Doing design research with youth at/from the margins in pandemic times: Challenges, inequalities and possibilities. In Kara

Khoo

S. M.

(Eds.), Researching in the age of COVID-19 volume 3: Creativity and ethics (pp. 116–123). Policy Press.

18.

Hallam

K. F.

(2021). Moving on from trials and errors: A discussion on the use of a forum as an online focus group in qualitative research. International Journal of Social Research Methodology, 00(00), 1–11. https://doi.org/10.1080/13645579.2021.1888402

19.

International Association for Public Participation (2018). IAP2 spectrum of public participation [internet]. https://cdn.ymaws.com/www.iap2.org/resource/resmgr/pillars/Spectrum_8.5x11_Print.pdf

20.

Khodyakov

Grant

Denger

Kinnett

Martin

Peay

Coulter

(2020). Practical considerations in using online modified-delphi approaches to engage patients and other stakeholders in clinical practice guideline development. The Patient, 13(1), 11–21. https://doi.org/10.1007/s40271-019-00389-4

21.

Kirwan

J. R.

de Wit

Frank

Haywood

K. L.

Salek

Brace-McDonnell

Bartlett

S. J.

(2017). Emerging guidelines for patient engagement in research. Value in Health, 20(3), 481–486. https://doi.org/10.1016/j.jval.2016.10.003

22.

Kite

Phongsavan

(2017). Insights for conducting real-time focus groups online using a web conferencing service. F1000Research, 6, 122. https://doi.org/10.12688/f1000research.10427.1

23.

Krueger

Casey

(2015). Focus groups: A practical guide for applied research (5th ed.). SAGE Publications.

24.

Laurance

Henderson

Howitt

P. J.

Matar

Al Kuwari

Edgman-Levitan

Darzi

(2014). Patient engagement: Four case studies that highlight the potential for improved health outcomes and reduced costs. Health Affairs, 33(9), 1627–1634. https://doi.org/10.1377/hlthaff.2014.0375

25.

Liberati

(2011). Need to realign patient-oriented and commercial and academic research. Lancet, 378(9805), 1777–1778. https://doi.org/10.1016/S0140-6736(11)61772-8

26.

Lobe

Morgan

Hoffman

K. A.

(2020). Qualitative data collection in an era of social distancing. International Journal of Qualitative Methods, 19(1), 1-8. https://doi.org/10.1177/1609406920937875

27.

Manafò

Petermann

Vandall-Walker

Mason-Lai

(2018). Patient and public engagement in priority setting: A systematic rapid review of the literaturee0193579. Plos One, 13(3). https://doi.org/10.1371/journal.pone.0193579

28.

McCarron

T. L.

Clement

Rasiah

Moffat

Wasylak

Santana

M. J.

(2021). Co-Designing strategies to support patient partners during a scoping review and reflections on the process: A commentary. Research Involvement and Engagement, 7(1), 1–8. https://doi.org/10.1186/s40900-021-00272-3

29.

Morgan

D.L.

(2019). Basic & advanced focus groups. SAGE Publications.

30.

National Institute for Health Research (2012). INVOLVE: Briefing notes for researchersNational Institute for health research. http://www.invo.org.uk/

31.

National Voices (2015). Prioritising person- centered care: the evidence [Inter- net]. London: National Voices; [cited 2014 Jul 30]. http://www.nationalvoices.org.uk/evidence

32.

Nelms

K. R.

Porter

A. L.

(1985). EFTE: An interactive Delphi method. 61.

33.

Nyumba

T. O.

Wilson

Derrick

C. J.

Mukherjee

(2018). The use of focus group discussion methodology : Insights from two decades of application in conservation. 2018(March 2017). https://doi.org/10.1111/2041-210X.12860, pp. 20–32.

34.

O ’Connor

Madge

Shaw

Wellens

(2011). Internet-based Interviewing In: The SAGE Handbook of Online Research Methods. 271–289. https://doi.org/10.4135/9780857020055

35.

Oravec

Arora

R. C.

Bjorklund

Gregora

Monnin

Duhamel

T. A.

Chudyk

A. M.

(2021a). Expanding enhanced recovery protocols for cardiac surgery to include the patient voice: A scoping review protocol. Systematic Reviews, 10(1), 1–10. https://doi.org/10.1186/s13643-020-01564-7

36.

Oravec

Arora

R. C.

Bjorklund

Gregora

Monnin

Duhamel

T. A.

Chudyk

A. M.

(2021b). Patient and caregiver-identified preferences and prioritized outcomes for cardiac surgery: A scoping review and focus group study. The Journal of Thoracic and Cardiovascular Surgery. In press http://doi.org/10.1016/j.jtcvs.2021.11.052

37.

Osborn

Squires

(2012). International perspectives on patient engagement: Results from the 2011 commonwealth fund survey. Journal of Ambulatory Care Management, 35(2), 118–128. https://doi.org/10.1097/JAC.0b013e31824a579b

38.

Pham

M. T.

Rajić

Greig

J. D.

Sargeant

J. M.

Papadopoulos

Mcewen

S. A.

(2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency. Research Synthesis Methods, 5(4), 371–385. https://doi.org/10.1002/jrsm.1123

39.

Phoenix

Nguyen

Gentles

S. J.

Vanderkaay

Cross

Nguyen

(2018). Using qualitative research perspectives to inform patient engagement in research. Research Involvement and Engagement, 4(1), 1–5. https://doi.org/10.1186/s40900-018-0107-1

40.

Richards

D. P.

Birnie

K. A.

Eubanks

Lane

Linkiewich

Singer

Begley

K. N.

(2020). Guidance on authorship with and acknowledgement of patient partners in patient-oriented research. Research Involvement and Engagement, 6(1), 1–8. https://doi.org/10.1186/s40900-020-00213-6

41.

Roberts

J. K.

Pavlakis

A. E.

Richards

M. P.

(2021). It’s more complicated than it seems: Virtual qualitative research in the COVID-19 era. International Journal of Qualitative Methods, 20, 1–13. https://doi.org/10.1177/16094069211002959

42.

Robinson

(1999). The use of focus group methodology Ð with selected examples from sexual health research. 29(4), 905–913.

43.

Rowe

Wright

(2001). Expert opinions in forecasting: The role of the Delphi technique. In Armstrong

J. S.

(Ed), Principles of forecasting. International series in operations research & management science (vol 30, pp. 125–144). Springer. https://doi.org/10.1007/978-0-306-47630-3_7

44.

Smith-MacDonald

Reay

Raffin-Bouchal

Sinclair

(2019). Patient-Oriented research and grounded theory: A case study of how an old method can inform cutting-edge research. International Journal of Qualitative Methods, 18(1), 1-13. https://doi.org/10.1177/1609406919863172

45.

Smythe

Giddings

L.S.

(2007). From experience to definition: Addressing the question ‘what is qualitative research? Nursing Praxis in New Zealand Inc, 23(1), 37–57.

46.

Staniszewska

Brett

Simera

Seers

Mockford

Goodlad

Tysall

(2017). GRIPP2 reporting checklists: Tools to improve reporting of patient and public involvement in research. BMJ (Online), 358, j3453. https://doi.org/10.1136/bmj.j3453

47.

Stocker

Brittain

Spilsbury

Hanratty

(2021). Patient and public involvement in care home research: Reflections on the how and why of involving patient and public involvement partners in qualitative data analysis and interpretation. Health Expectations(April), 1–8. https://doi.org/10.1111/hex.13269

48.

O’Leary

Nagraj

El-Awaisi

O’Carroll

Xyrichis

(2020). Doing interprofessional research in the COVID-19 era: A discussion paper. Journal of Interprofessional Care, 34(5), 600–606. https://doi.org/10.1080/13561820.2020.1791808

49.

Teti

Pichon

Myroniuk

T. W.

(2021). Community-engaged qualitative scholarship during a pandemic: Problems, perils and lessons learned. International Journal of Qualitative Methods, 20(1), 1-4. https://doi.org/10.1177/16094069211025455

50.

Teti

Schatz

Liebenberg

(2020). Methods in the time of COVID-19: The vital role of qualitative inquiries. International Journal of Qualitative Methods, 19(1), 1-5. https://doi.org/10.1177/1609406920920962

51.

Thornton

(2014). Beyond rhetoric: We need a strategy for patient involvement in the health service. BMJ (Online), 348, g4072. https://doi.org/10.1136/bmj.g4072

52.

Tinetti

M. E.

Basch

(2013). Patients’ responsibility to participate in decision making and research. JAMA - Journal of the American Medical Association, 309(22), 2331–2332. https://doi.org/10.1001/jama.2013.5592

53.

Vat

L. E.

Finlay

Jan Schuitmaker-Warnaar

Fahy

Robinson

Boudes

Broerse

J. E. W.

(2020). Evaluating the “return on patient engagement initiatives” in medicines research and development: A literature review. Health Expectations, 23(1), 5–18. https://doi.org/10.1111/hex.12951

54.

Zoom Video Communications (2021). Zoom for healthcare. https://zoom.us/healthcare

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.16 MB

0.00 MB

A Virtual,Multi-Session Workshop Model for Integrating Patient and Public Perspectives in Research Analysis and Interpretation

Abstract

Keywords

Introduction

Method

Contextualizing Model Development: Introducing Our Scoping Review Case Example

Our Research Team

Patient and Public Involvement

Recruitment into the Virtual Consultation Workshop

Model for Integrating Patient and Public Perspectives in Research Analysis and Interpretation

Session 1: Familiarization with the Virtual Platform

Key Considerations

Application in Case Example

Overview of the Structure of Sessions 2–4

Session 2

Key Considerations

Post-Session Analyses

Application in Case Example

Session 3

Key Considerations

Post-Session Analyses

Application in Case Example

Session 4

Key Considerations

Application in Case Example

Other considerations

Equity, Diversity, and inclusion

The Virtual Platform

Research Team Member Role Descriptions

Participant Feedback

Discussion

Conclusions

Supplemental Material

Supplemental Material - A Virtual, Multi-Session Workshop Model for Integrating Patient and Public Perspectives in Research Analysis and Interpretation

Footnotes

Declaration of Conflicting Interests

Funding

Research Ethics and Patient consent

ORCID iD

Supplemental Material

References

Supplementary Material