Abstract
What is Qualitative Evidence Synthesis?
A qualitative evidence synthesis, or QES, is a type of systematic review that brings together the findings from primary qualitative research in a systematic way. A primary qualitative research study is one that uses a qualitative method of data collection and analysis. Sometimes if there is a lack of primary qualitative research studies, then qualitative data can be used, for example from open ended questions in questionnaire studies. Evidence from a primary qualitative study is however likely to be conceptually richer and thicker in description, and has the potential to make a bigger contribution to a qualitative evidence synthesis (Noyes, Booth, Cargo, Flemming, Harden, et al., 2019). The aim of a QES is to establish a greater understanding of the kind of issues, often of a subtle or sensitive nature, that primary qualitative research frequently addresses. The findings from a QES can provide rich interpretations relating to the impact of a condition and can enable a greater understanding of individuals’ and groups’ experiences, views, beliefs and priorities for healthcare (Flemming et al., 2019).
The term QES is used, and is the preferred term of the Cochrane Qualitative and Implementation Methods Group, as it acknowledges that qualitative research requires its own methods for synthesis which reflects the nature of the qualitative paradigm, rather than simply using the same methods devised for systematic reviews of quantitative research (Booth et al., 2016). However, the terminology around QES can be confusing as it is an umbrella term for a number of approaches to qualitative synthesis; details are provided in the glossary of terms maintained by the Cochrane Qualitative and Implementation Methods Group https://methods.cochrane.org/qi/contact-us (Online Appendix 1). Other terms include (Booth et al., 2016): Qualitative systematic review Qualitative meta-synthesis Qualitative research synthesis
QES methods were developed in response to an increasing demand from health professionals, policy makers, guideline developers and educationalists for review evidence that goes beyond “what works” in a specific context. QES methods are able to address additional questions that complement those traditionally answered through systematic reviews of quantitative evidence, particularly reviews of randomized controlled trials (RCT) (Flemming & Jones., 2020)
Through a QES, evidence is synthesized from primary qualitative studies with the aim of developing new cumulative knowledge. This differs to a more traditional literature review of qualitative research which seeks to combine studies in a summary format (Flemming & Jones, 2020). Depending on the QES method selected, the process can enable researchers to “go beyond” the individual findings of studies, and produce something greater than the sum of the individual parts (Carroll, 2017). In doing this, findings may be identified that are not seen as important in a single qualitative study, and more powerful explanations can be made (Carroll, 2017).
The methods for QES can also facilitate the integration of synthesized findings from qualitative research with systematic reviews of effectiveness of interventions; methods are developing for doing this. Such syntheses can help to increase understanding of a particular phenomenon; help identify associations between the environment in which people live and the implementation of an intervention; help develop understanding of health conditions and the interventions that treat them from the perspective of those with the condition, or who treat people with it; and help understanding of the complexity of interventions and implementation, and their impacts and effects on different subgroups of people and the influence of individual and contextual characteristics within different contexts. (Noyes, Booth, Cargo, Flemming, Harden, et al., 2019).
There are now over 30 methods for conducting a qualitative evidence synthesis, and although these methods have evolved substantially over the last decade, some methods have been subject to more development and testing than others (Noyes, Booth, Cargo, Flemming, Garside, et al., 2018). Therefore, the choice of method used is critical to the success of the synthesis. This paper provides information to support methodological decision making by detailing the most commonly used QES methods with the greatest number of exemplars.
What Kind of Questions Can a QES Explore?
Questions that seek to enhance understanding of a particular phenomenon of interest e.g. understanding individual’s experiences of living with urinary incontinence (Toye & Barker, 2020);
Questions that increase our understanding of the values and attitudes toward, and experiences of, health conditions or interventions by those who implement or receive them e.g. exploring the factors that affect the implementation of strategies to substitute doctors with nurses in primary care (Karimi-Shahanjarini et al., 2019);
Those that identify associations between the broader environment within which people live and the interventions that are implemented e.g. examining older people’s experiences of everyday travel within the urban environment (Graham et al., 2020)
providing a detailed understanding of the complexity of interventions and implementation, and their impacts and effects on different subgroups of people and the influence of individual and contextual characteristics within different contexts e.g. developing an understanding of the factors that influence the success and sustainability of lay health workers across different health care settings and contexts (Glenton et al., 2013)
Processes Associated With a Qualitative Evidence Synthesis
Question Formulation
Developing a review question for a QES is an important step that carries a number of considerations. Qualitative evidence syntheses ask “how and why questions,” meaning the questions formulated are exploratory in nature and aim to identify what is known about a phenomenon from one or more perspectives. This exploratory process means that the initial review question may be quite broad with the aim of mapping what is known already. The scope of review will need to be determined prior to the review question being formulated (Harris et al., 2018). “Scope” refers to the kind of boundaries that will exist around the review, framing the topic of interest and mapping the existing information available (Harris et al., 2018). This is one of the key differences to question formulation for a quantitative review, that in a QES, while the questions can be fixed from the start of the review, also may emerge as a result of the findings from the initial review process (Booth et al., 2016). It can be helpful to think of the question as either an “anchor” ie fixed at the start of a review, or as a “compass” that guides the review (Eakein & Mykhalovskiy, 2003).
There are a number of structures that have been developed to support the development of a research question for a QES. Review questions for quantitative reviews are commonly mapped using a variant of the PICO (Population, Intervention, Counter-intervention, Outcome) tool. The different aims and foci of a QES mean this particular structure often doesn’t fit the nature of the review question being asked by a QES. Due to the scope of research questions that can be answered by a QES alternatives commonly include some consideration of the context (i.e. setting, context or environment) of the question being asked (Harris et al., 2018), in the form of “setting,” “perspective,” “phenomenon of interest,” the following two structures are often used. SPICE (Setting, Perspective, Intervention or Phenomenon of Interest, Comparison, Evaluation) (Booth., 2006b) SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type); (Cooke et al., 2012).
More recently the structure “PerSPecTIF” (Perspective, Setting, Phenomenon of interest/ Problem, Environment, Comparison (optional), Time/ Timing, Findings) has been developed to extend QES question formulation to describe both the wider context and immediate setting, components that are particularly suited to qualitative evidence synthesis (Booth et al., 2019). Below is a worked example of PerSPecTIF (James, 2020) (Table 1) for a synthesis exploring provision of palliative care for people who are homeless or vulnerably housed from the perspective of the individuals themselves and those who care for them.
A Worked Example of PerSPE(C)TiF.
Use of a tool such as PerSPecTIF may result in questions addressing aspects of feasibility and acceptability, in the way other QES question formulation structures may not. In turn, this can lead to more informed decisions on choice of synthesis method and greater consideration of context within a review (Booth et al., 2019).
Finally, when developing the focus of a QES, stakeholders should always be involved in framing the issues and developing the question structure in order to explore “What,” “For whom” and “Why” and in “What context” the focus of the review exists (Flemming et al., 2019).
Protocol Development
Few sets of guidance for QES focus on the development of a protocol, but this is a key step in a QES and is closely tied to the process of focusing its topic and question. The writing of a protocol enables the development of the case for the importance of the review and why a synthesis of qualitative evidence is relevant to the question being posed. The nature of the question informs both the searching, the criteria for inclusion and methods chosen to undertake the review and the protocol formalizes these processes; the coherence of the protocol formalizes the review’s credibility (Harris et al., 2018). It is worth bearing in mind however that the iterative nature of the development of the focus of a review can lead to a protocol needing to state an open and flexible approach to the review process (Booth et al., 2016). For a QES that has a health related focus/outcome, it can be registered on the PROSPERO international database of prospectively registered systematic reviews (https://www.crd.york.ac.uk/prospero/).
Searching for Literature and Inclusion and Exclusion Criteria
Searching for qualitative research for inclusion in a QES remains an area of ongoing methodological development. Key guidance around the processes for searching and inclusion stem from the “7 S” approach (Booth et al., 2016). These seven steps ask the reviewer to consider issues of: Sampling of papers—it needs to be considered whether this should be comprehensive (include everything) or purposeful or theoretical (when the intent is to generate an interpretative understanding) (Suri, 2011); decisions need to be justified and matched to the focus of the review. Different sampling strategies may be driven by epistemological approaches to a review. Pragmatically however, review teams need to ensure, if they choose an approach to study selection that includes sampling, the underlying theoretical perspective is stated alongside a description of methods used and the rationale that underpins this. (Booth, 2016). A worked example of purposive sampling is provided by Ames et al. (2019)
Sources—the databases searched need to reflect to the scope and topic under review. For health-related questions MEDLINE and the Cumulative Index of Nursing and Allied Health Literature (CINAHL) may suffice, with other more topic focused databases if required Structured questions—using an appropriate format as discussed above Search procedures—should generally privilege specificity (retrieval of only relevant items) over sensitivity (retrieval of all potential items) as qualitative research is generally less prevalent that quantitative research. A caveat here is that particular challenges exist in retrieving qualitative research because of non-informative titles and abstracts, diffuse terminology and poor indexing. This may require additional searching of supplementary sources to overcome these limitations Search strategies and any methodological filters used should match the purpose of the review. A simple three-line qualitative filter using the terms “qualitative,” “findings” and “interviews” may help improve retrieval of more relevant items (Flemming & Briggs, 2007) Supplementary strategies include reference checking, citation searching, handsearching of particular journals and contact with authors or subject experts to identify key missing papers Standards for reporting searching—include both the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) flowchart (Moher et al., 2010) and the STARLITE mnemonic (sampling, strategy, type of study, approaches, range of years, limits, inclusion and exclusions, terms used, electronic sources) (Booth, 2006a)
Assessing Methodological Limitations
The issue of whether to undertake a quality assessment of the methodological strengths and limitations of a primary qualitative study for potential inclusion in a qualitative evidence synthesis remains a contentious one, with often divided opinion (Garside, 2014, Noyes, Booth, Flemming, Garside, Harden, et al., 2018). Equally contentious is what to do with the assessment once it has been conducted. Overall, some form of quality assessment is commonly undertaken in a QES, and there is some expectation from journal editors that this will be presented as part of a submitted manuscript. Whether an assessment of methodological limitations is undertaken or not, a justification needs to be presented as to the chosen approach.
Undertaking an assessment of the methodological limitations of primary qualitative studies can however provide useful information to inform decisions when conducting the QES. Where an assessment is made, qualitative researchers generally identify methodological strengths and limitations of the primary studies included in the synthesis ie an appraisal of “rigor.” This assessment can help for example with sampling decisions as well as determining the whether the data contained within studies is conceptually rich or not or descriptively thick or thin and what type of synthesis method is the best fit (Noyes, Booth, Cargo, Flemming, Harden, et al., 2019). One of the most commonly used tools to do this is the Critical Appraisal Skills Programme (CASP) tool for qualitative studies. A set of domains has been recommended that have evolved from extensive practice (rather than empirical study) that should be considered when assessing methodological limitations of a study (Noyes, Booth, Flemming, Garside, Harden, et al., 2018): Clear aims and research question Congruence between the research aims/question and research design/method(s) Rigor of case and/or participant identification, sampling, and data collection to address the question Appropriate application of the method; richness/conceptual depth of findings, exploration of deviant cases and alternative explanations, and reflexivity of the researchers.
The CASP tool for qualitative studies maps onto the domains above.
Data Extraction and Methods of Synthesis
Data extraction in a QES is a two-stage process. Firstly, it is important to extract the “contextual” details eg. the population studied and their characteristics, the context in which the study occurred, methodology and methods used in recruitment, data collection and analysis and to record these in a table of included studies (Noyes, Booth, Flemming, Garside, Harden, et al., 2018). This should occur irrespective of the approach being used for the review, and can occur at the same time as quality appraisal as much of the information required is the same for both processes. Knowing the details of the participants included in the individual studies and their context is central to a user of a review being able to interpret the findings (Flemming & Jones, 2020).
The second stage of data extraction is the extraction of the “findings” from the individual primary studies. “Findings” in a qualitative study predominantly take the form of quotes from participants, author interpretations, themes and sub-themes, new theory or observational excerpts. Commonly these are presented in a narrative within a paper, but may also appear as tables, infographics, logic models etc (Noyes, Booth, Flemming, Garside, Harden, et al., 2018). Findings tend to appear (unsurprisingly) within the “Results” or “Findings” section of a paper, but author interpretation may also occur within the “Discussion” section of a paper, depending on journal format requirements. It is also common to find “findings” in the abstract, summary statements and additional online only files. Information about theory and the theoretical frameworks through which data has been analyzed can also be found in the methods section.
The key principle of data extraction in a QES, and for the later processes of analysis and synthesis, is that it is not a one-off, sequential, linear step. Typically, data extraction, analysis and synthesis are iterative phases, involving movement backward and forward between them (Flemming & Jones, 2020). In this sense, describing these stages as separate steps within an article such as this falsely delineates them, however for those new to QES can help provide guidance.
There are a variety approaches to extracting data and how they are managed once extracted. Many reviewers export the data into some form of qualitative data management system such as NVIVO or Atlas-Ti; similar to how textual data would be managed in primary qualitative research. Such systems enable the management of large volumes of text and have functionality that support the organization and analysis of text. With smaller amounts of data or where access isn’t available to specialist systems, data can be managed within word processing or spreadsheet software.
Ultimately the methods for qualitative data extraction vary according to the chosen method of synthesis. It is important that the appropriate method of data extraction is used with a specific method of qualitative evidence synthesis. Different methodologies are designed to have different outcomes; some lead to descriptive level findings eg Meta-aggregation (Hannes & Lockwood, 2011), while others can lead to the development of new theory eg Meta-ethnography (Noblit & Hare, 1988). It is increasingly advocated that the choice of synthesis method should not be finally determined until the group of included papers is established and the reviewers know the type of data contained within them (Noyes, Booth, Cargo, Flemming, Garside, et al., 2018). Because of this, flexible options concerning choice of method may need to be articulated in the protocol.
There are around 30 different methodologies and methods that can be used when undertaking a QES that are in various levels of development and sophistication (Booth et al., 2016). While we have provided detail for the three most commonly used QES methods below, full guidance is available elsewhere (Booth et al., 2016). In summary, the domains in the RETREAT guidance are those that any reviewer needs to take into account prior to starting their review (Booth et al., 2018) (Table 2).
Domains for Consideration for Choice of Approach for a QES as per RETREAT Guidance (Reproduced From Booth et al. (2018) doi.org/10.1016/j.jclinepi.2018.03.003).
It is important to bear in mind that while many QESs are undertaken as stand-alone reviews, a QES can also be undertaken with the aim of integrating it with a review of effectiveness. While methods for integration are less well developed than methods for synthesizing qualitative research, there has been increasing interest in doing this over the last few years and good exemplars now exist (Harden et al., 2018). While it is not the aim of this paper to provide detailed explanation as to how to do this, two out of the three QES methods outlined below can also be used when undertaking an integrative review.
These three methods; thematic synthesis, framework synthesis (or best fit framework synthesis), and meta-ethnography are some of the most developed methods for QES and while there are similarities between them, each provides a unique approach to a QES and has advantages and disadvantages (Table 3). While an outline is given for all three approaches here, we recommend that you refer to the original texts that describe the methods and associated references below to gain further understanding of them, alongside the guidance of choice of method by Booth et al. (2016) & the RETREAT guidance (Booth et al., 2018).
Recommended Methods for Undertaking a Qualitative Evidence Synthesis
* Method choice depends ultimately on the pool of evidence available.
Thematic Synthesis
Thematic synthesis is an interpretative approach to reviewing based on the methods of thematic analysis used in primary research. Thematic synthesis methods however go further than thematic analysis methods and enable new insights, interpretations and theory to be developed not seen in individual primary studies. It is a frequently used method and good for novice reviewers due to its straight forward approach. It also has flexibility as to the type of data from primary research that can be included in a review through its methods, allowing the incorporation of both “thin” and/or “thick” data in the development of analytical themes through an inductive approach to coding (Thomas & Harden, 2008). Thematic synthesis, briefly, involves a three-staged approach, starting with the line-by-line coding of the findings of the individual studies to identify potential areas of similarity that may be developed into descriptive themes. Depending on the type of data in the primary studies, the thematic synthesis may end here, however if the data are rich enough, then the reviewer can go beyond the descriptive themes and develop analytical themes which aim to generate new constructs, explanations or hypotheses (Heyvaert et al., 2016).
Thematic synthesis is particularly helpful approach for novice reviewers as it prescribes an organized and structured way of developing primary data into prominent descriptive and analytical level themes. A downside of this approach is that it can become a simplistic, descriptive account of the frequency of themes rather than producing a higher level of explanation, especially if the reviewers lack experience in qualitative analysis and synthesis or run out of time to complete the method as intended (Heyvaert et al., 2016).
How Has It Been Used?
Thematic synthesis is one of the most commonly used methods of qualitative evidence synthesis and has been used as an approach to address a wide-range of topics and questions, particularly those that seek to describe the range of peoples’ beliefs, attitudes, expectations, and experiences of health care, disease and illness, health care interventions etc. Examples include: Exploring patient expectations and experiences of remote monitoring for chronic diseases (Walker et al., 2019) Examining the research burden of randomized controlled trial participation (Naidoo et al., 2020). Identifying the concerns of people with advanced illness experiencing breathlessness to guide clinical assessment and outcome measurement (Lovell et al., 2019).
Framework Synthesis, or Best-Fit Framework Synthesis
As with thematic synthesis, the origins of framework synthesis are based in a primary research method—framework analysis. Framework synthesis offers a highly structure approach to QES by using an apriori framework, into which the findings from the primary qualitative research are extracted and synthesized (Booth et al., 2016); in this way it is distinct from the other two methods described in this paper. It is predominantly deductive in its approach and, although the generation of theory can be an outcome for framework synthesis, its main function is to interpret and integrate what is happening within a particular setting (Flemming et al., 2019). Framework synthesis is therefore highly suitable for applied policy or clinical questions in a specific setting or context.
Framework synthesis is best used when an existing framework can be applied to the review. Frameworks can derive from a pre-existing review, from a conceptual model, from a policy framework or from a logic model and in this sense, the concepts that drive the synthesis are “secure” beforehand. The chosen framework can also become a scaffold on which both quantitative and qualitative data can be juxtaposed, making it a suitable choice for reviews of incorporating both qualitative and quantitative research (Flemming et al., 2019).
Best Fit Framework Synthesis
Best fit framework synthesis is a version of framework synthesis that draws on the advantages of both framework synthesis and thematic synthesis in as much as its starting point is a framework that is “good enough.” This framework is populated by the data from the primary studies in a deductive way as per framework synthesis however without the need to force data that don’t “fit” into unsuitable categories. The remaining data are synthesized inductively using the approaches of thematic synthesis to develop themes until all the data are accounted for (Booth et al., 2016). Therefore, there is an explicit two-stage sequential process to best fit framework synthesis which enables an audit trail of themes arising from the framework synthesis and those from the subsequent thematic synthesis (Booth et al., 2016).
It is also possible in a framework synthesis for new topics to be identified and incorporated as they emerge from the data, allowing the development of a “best fit” model that can be enhanced by the addition of new findings arising from the synthesis of a broader body of literature (Booth et al., 2016).
How Has It Been Used?
Framework synthesis is currently the most commonly used approach when the review is part of a guideline process and works well when the focus of a review requires an understanding of complexity around feasibility and health system considerations (Flemming et al., 2019). Examples include: Establishing the barriers and facilitators experienced when implementing lay health worker programmes to improve access to maternal and child health (Glenton et al., 2013) Explore and explain health outcomes of online consumer health information in primary care in order to help patients to find and use relevant understandable information (Pluye et al., 2019) Examining the leadership and management competencies for hospital managers in order to develop a leadership and management competency framework for health service managers (Kakemam et al., 2020)
Meta-Ethnography
Meta-ethnography is an interpretative approach to synthesis which aims to create new understandings and theories from a body of work. It is one of the longest standing methods for QES having been developed in the late 1980s specifically for the purpose of synthesizing primary qualitative research (Noblit & Hare, 1988). The purpose of a meta-ethnography is to bring together primary qualitative research in the form a whole that contributes something new and above the individual studies’ findings. This is very different from an aggregative approach to synthesis, aiming instead to develop comparative understanding (Heyvaert et al., 2016).
Noblit and Hare outlined a seven-stage process when undertaking a meta-ethnography, although this is not linear and involves movement between the stages as the review process proceeds. The steps are outlined below (Atkins et al., 2008, Flemming et al., 2013): Step 1: Getting started
This first phase involves determining a focus for the review that can be informed by a synthesis of qualitative research. Step 2: Deciding What is Relevant to the Initial Interest
The second step in the process involves further defining the focus of the synthesis, often through an iterative process of an early examination of the existing body through preliminary searches, through this refining the question, further locating relevant studies; making decisions on inclusion; and quality assessment. Step 3: Reading the studies
In this step studies are read to develop an understanding of their position and context before being compared with others. This often involves repeated re-reading of studies to identify key findings. At this point an appraisal of methodological limitations and data extraction can occur. Step 4: Determining how the studies are related
In determining how the studies are related, Noblit and Hare suggest looking at the relationships between individual studies by compiling a list of the key findings in each study and comparing them with those from other studies. If the findings of studies oppose one another, Noblit & Hare advise a form of synthesis called “refutational synthesis” can be undertaken Step 5: Translating studies into one another
Translating the studies into one another is a key stage of meta-ethnography. “Translating” is a synthesis term particular to meta-ethnography and which involves comparing the similarities and differences of key findings in one study with those of others and translating (ie integrating or synthesising) them into one another, in essence, to produce themes (Atkins et al., 2008). Ultimately the translations represent a reduced account of all studies. This is the first level of synthesis and is called reciprocal translation. Step 6: Synthesizing translations
This is stage in which the higher level of interpretation associated with a meta-ethnography occurs. In the same way a primary study might move from descriptive to explanatory analysis, a meta-ethnography can proceed from reciprocal translation to a higher order interpretation which distils the translations into more than the parts alone imply. This is termed a “line of argument” synthesis and is the second level of synthesis. Step 7: Expressing the synthesis
The final step requires consideration on the type of format the synthesis should be reported in. For many this will be in the form of a report or academic journal, or for post-graduate students a thesis or chapter of their dissertation.
How Has It Been Used?
Meta-ethnography has predominantly been used to achieve greater interpretation and depth of understanding of more complex phenomenon in health and social care and education. To explore the experiences of care delivered telemedicine for people with chronic obstructive pulmonary disease (Barken et al., 2019) Examining how medical education can affect empathy and compassion in medical students (Krishnasamy et al., 2019) Exploring the experiences of people living over the age of 80 (Toye et al., 2020)
Determining the Confidence in the Findings From a QES—Grade CERQual
The GRADE CERQual (confidence in the evidence from reviews of qualitative research) was developed to make the findings more accessible and understandable for decision makers (Lewin et al., 2018). CERQual provides the user with an assessment of how much confidence to place in individual QES findings. QES reviewers are increasingly including GRADE CERQual assessments in their reviews as a marker of best practice. CERQual provides a transparent and systematic framework for assessing confidence in individual review findings, based on assessment of four components: (1) methodological limitations, (2) coherence, (3) adequacy of data, and (4) relevance. A fifth component, dissemination (or publication) bias, may also be important and is being explored.
Detailed guidance on applying GRADE CERQual can be found at the following link: https://implementationscience.biomedcentral.com/articles/supplements/volume-13-supplement-1
Reporting a QES
Reporting guidelines help drive the standards for reporting for all types of research methodologies and qualitative evidence syntheses are no exception. However, producing consolidated guidance across the approaches to QES approaches is challenging because of the broad variety of paradigms, approaches, designs and techniques that are available. Such variety is no bad thing as it offers good grounds for methodological debate and as a result, methodological progress. Currently there are a small number of different tools for reporting individual aspects of a synthesis eg the reporting of methods of searching (Flemming et al., 2018) alongside one generic reporting tool for a QES available, called “Enhancing transparency in reporting the synthesis of qualitative research” (ENTREQ) (Tong et al., 2012). There are also two methodologically specific tools, one of which provides guidance on reporting of meta-ethnographies—eMERGe (France et al., 2019).
The ENTREQ statement was developed to promote explicit and comprehensive reporting of the synthesis of qualitative studies and its purpose is to offer guidance for researchers and reviewers to improve the reporting of syntheses of qualitative health research. It consists of 21 items which a reviewer should look to report which are contained within five overarching domains: Introduction, methods and methodology (Domains 1 and 2)
While it is a generic tool, the ENTREQ tool documents the most frequently used methods for qualitative evidence synthesis to which it might apply, acknowledging that the approaches and methodology for synthesis are usually driven by the posed research questions.
Since the development of ENTREQ, other methodologically specific reporting guidelines have been developed. Of specific relevance to this paper is a guideline for reporting meta-ethnographies called eMERGe (France et al., 2019). It was developed with the intention to improve the clarity and completeness of reporting of meta-ethnographies to facilitate the use of their findings to support health and social care policy and practice (France et al., 2019).
The eMERGe guidance contains 19 reporting criteria grouped into separated into seven phases, reflecting the seven steps of a meta-ethnography: Phase 1—Selecting meta-ethnography and getting started Phase 2—Deciding what is relevant Phase 3—Reading included studies Phase 4—Determining how studies are related Phase 5—Translating studies into one another Phase 6—Synthesizing translations Phase 7—Expressing the synthesis
One further methodologically specific reporting tool is available in the form of the RAMESES statements for reporting other types of syntheses that may incorporate primary qualitative research. The RAMESES statements support the reporting of Realist Reviews (Wong et al., 2013a) and meta-narrative reviews (Wong et al., 2013b). These are synthesis methods more suited to providing additional context to a heterogeneous topic area by the inclusion of both qualitative and quantitative research in a review (meta-narrative review) or as an application for implementation research (Realist Review).
A 5-point “decision flow-chart” is available to help review authors in their decision making over the use of a reporting tool. The flowchart highlights that decisions around reporting are dependent on “
Author Reflexivity
As with any form of qualitative inquiry, it is important to consider any influences or biases that the review team may hold, how these potential threats to rigor are handles and any potential impact on the interpretation of findings. The key principle is to be transparent and to consider any potential conflicts of interest carefully as a team and to record them in a public protocol. The sorts of aspects that require documenting in relation to author reflexivity include the following: The funder and whether they had any involvement in conducting the review and in particular whether they had any influence on developing or editing the findings. The composition of the review team and any relevant positions or beliefs held concerning the review question and phenomenon of interest that could influence the way that the evidence was interpreted Conflicts of interest, including financial and non-financial (eg relationships with key people who could potentially exert influence on the development of findings). Team governance procedures and processes to maintain internal validity (for example, when selecting studies, conducting quality appraisal, data extraction and coding, undertaking the synthesis, developing and finalizing the findings and developing new theory) Protocols for processing evidence when one of the review authors is also an author of a primary qualitative study of interest Ways of working and engaging with key stakeholders to ensure that no undue influence occurs
Role of Key Stakeholders and Consumers
It is a marker of best practice to engage with key stakeholders, consumers and patient and public representatives to ensure that the QES is developed with multiple actor perspectives and the findings are grounded in reality. It is common to have consumers and patient and public representatives as members of the review team from inception to dissemination to ensure for example that the review question is relevant and the phenomenon of interest are written with various perspectives in mind. Other opportunities for wider involvement include membership of steering and advisory groups to ensure appropriate governance and engagement with the wider community of consumers and patient and public representatives.
What Does a “Good/Rigorous” QES Looks Like?
A well conducted and methodologically rigorous QES is surprisingly rare. When you do see one it is an absolute pleasure to read—see for example Toye et al., 2013, Gomersall et al., 2011. At present there is no tool to assess the rigor of a QES report that has undergone thorough evaluation and testing. There are three tools which are currently available as follows: A QES checklist has been developed by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU) Sweden: https://www.sbu.se/contentassets/14570b8112c5464cbb2c256c11674025/methodological_limitations_qualitative_evidence_synthesis.pdf
A further tool for appraising QES reviews that is under development is the MACACQUES tool used by the Evidence Synthesis of Qualitative Research in Europe (ESQUIRE) Methods Workshop https://tinyurl.com/macaquesQA
For those undertaking a mixed methods synthesis there are principles of good practice for Mixed available written by Jinemez et al. (2018) https://doi.org/10.1080/19439342.2018.1534875 To link to this article:
It is however sometimes easier to articulate what a poor and not rigorously conducted QES looks like. Table 4 outlines some of the common methodological issues found in QES reports.
Common Methodological Issues in Qualitative Evidence Synthesis Reports.
Discussion and Conclusion
The paper has outlined the purpose of undertaking a QES, has provided details as to how to undertake each of the stages of a QES, alongside providing considerations for the overall approach to a QES. As such the paper presents the “state of the methods” for qualitative evidence synthesis, signposting to contemporary methodological developments and approaches. It seeks to demonstrate approaches to synthesis that will enable the reviewer to develop an understanding of the phenomena under study that goes beyond that possible in a more traditional literature review.
The paper is written in the acknowledgment that QES is not without its critics. There are researchers and reviewers who consider that the newer methodological developments over the last few years represent a move away from the aim of qualitative evidence synthesis to ‘expand insights about complex human phenomena” (Thorne, 2017, p. 3). There is concern that newer approaches have moved QES in a direction that is more technical methodologically and superficial theoretically (Thorne, 2017). New methodologies do, however, need to be used and validated, in order to advance the field.
There are those who consider that primary qualitative research should only be used in the context in which the original data were collected and it should not be taken away from that context (Pope & Mays, 2006). Those who pursue QES and its methodological development tend to adopt a position in that acknowledges that research studies attempt to describe and capture (albeit in different ways) an underlying social reality (Flemming, 2007). We propose both a relatively pragmatic approach to synthesis; for those concerned with answering questions relevant to clinical practice and policy decision making, adopting this pragmatic stance makes the synthesis of qualitative research methodologically feasible. We also suggest there is an ethical imperative to systematically synthesize existing qualitative research. In doing so we can establish what is known about a particular area or phenomenon of health care, which, in turn, can help develop the focus of any new piece of primary research, thus preventing replication and the recruitment of individuals to studies when the answer is already known.
Footnotes
Acknowledgments
Declaration of Conflicting Interests
Funding
ORCID iD
