The Multi-Stage Mixed Methods Framework: A new research design to combine hypothesis development and hypothesis testing and to embed machine learning and practitioner engagement in the social sciences

Abstract

Multi-methods research designs typically focus either on developing or on testing pre-existing hypotheses. We outline a new methodological framework to combine hypothesis development and testing into a coherent and robust multi-method research design: the Multi-Stage Mixed-Methods Framework (MSMMF). The MSMMF is a novel approach to carefully sequence and combine different methods, such as machine learning (ML), practitioner engagement, inferential statistical analysis, qualitative comparative analysis, process-tracing and/or congruence analysis. We demonstrate that the MSMMF provides a holistic research design for developing and testing hypotheses, combining the strengths of existing mixed-methods approaches and embedding ML and the involvement of practitioners throughout the research process. We present the MSMMF’s application to a theoretically challenging, empirically rich and policy-relevant question: Why do some peace processes bring an end to large-scale conflict-related violence while others do not?

Introduction

In the field of civil war recurrence, as elsewhere in the broader social sciences, an ever-growing amount of methodologically robust empirical data underpin a wealth of individually rigorous but cumulatively inconclusive studies. Fragmented findings reflect competing theoretical explanations, multiple causal mechanisms, different definitions and operalizations of concepts (Neudorfer et al., 2025), competing measurements, variety in the case studies and/or the contextual factors considered. As a result, it is becoming increasingly challenging to develop and test comprehensive hypotheses, formulate general theories and produce coherent policy recommendations.

In this paper, we offer a novel framework that combines the methodological strengths of existing multi-methods research designs. It allows researchers to both develop and test hypotheses by harnessing the foundational principles of artificial intelligence, engaging users and practitioners at all stages of research and building on the proven strengths of existing qualitative and quantitative methods. Our Multi-Stage Mixed-Methods Framework (MSMMF) is a first step in addressing the needs of scholars in the ever more complex social sciences, and more specifically in peace and conflict studies.

The MSMMF builds on three significant methodological advancements in the social sciences to provide three sets of innovations. First, valuable mixed-methods research designs have emerged to overcome a persistent divide between qualitative and quantitative approaches in fields where, ‘because of the [their] different fundamental assumptions, it is very difficult for in-depth studies of individual cases to communicate meaningfully with claims about mean causal effects across a large set of cases’ (Beach, 2020: 163). Despite extensive progress, existing mixed-methods frameworks often remain skewed towards one end of the methodological spectrum, emphasizing either qualitative or quantitative techniques and reproducing ‘a larger methodological divide than commonly understood’ (Beach, 2020: 164). They also typically focus either on exploratory pattern recognition (hypothesis development) or explanatory pattern testing (hypothesis testing) (Goertz, 2017; Humphreys and Jacobs, 2015; Lieberman, 2005; Seawright, 2016). As a result, they are not ideal to explore research puzzles focusing on causes-of-effects, where a combination of hypothesis development and testing is necessary.

While recognizing the usefulness and value of existing methodological tools, we propose that some research questions demand different research frameworks that combine the strengths of existing qualitative, quantitative and mixed-methods approaches, to both develop and test hypotheses. We developed the MSMMF to sequence qualitative and quantitative research methods while combining the development of hypotheses (exploratory pattern recognition) and their testing (explanatory pattern testing) and providing opportunities for researchers to employ both small-n case-based (qualitative) and variance-based (quantitative) techniques at both stages. Thus, the MSMMF bridges the gap ‘between what can be termed a “bottom-up” case-based approach that focuses on tracing how causal mechanisms play out in individual cases, and a “top-down” variance-based approach that assesses the mean causal effect of variables within a population’ (Beach, 2020: 164). The MSMMF also empowers researchers with increased methodological agency because it encourages feedback loops between hypothesis development and testing, and more opportunities to mediate between interim findings generated by distinct methods at different stages of the research design.

Second, the expansion and refinement of computational social sciences present unique opportunities for identifying general trends and themes in high-dimensional datasets, in which variables by far outnumber observations (Grimmer et al., 2021). The MSMMF adds machine learning (ML) into mixed-methods research designs. Previously overlooked in multi-methods research, we show that supervised ML can be combined with qualitative research techniques to develop robust hypotheses in fragmented and contradictory research fields. These hypotheses can subsequently be tested through appropriate statistical methods and explored through in-depth qualitative case studies.

Third, academics have increasingly engaged with policymakers, practitioners and potential research users to disseminate their research findings, among others, through in-person talks, workshops, consultations and podcasts. Policy engagement has been encouraged by funding bodies and research quality assessment exercises in countries such as the United Kingdom and Germany. We propose embedding engagement with practitioners and users at various stages of research, transforming them from recipients to research participants. In our field, we define practitioners as individuals who actively engage in the practical application of conflict prevention, mediation and peacebuilding. They are potential research users because they can employ our research findings to inform their activities, but they can also contribute to identify and refine research questions and evaluate the relevance of insights (Bobo et al., 2024). The MSMMF recognizes the crucial contribution that practitioners can make to academic insights (and vice versa) by embedding iterative opportunities for user engagement in all stages of research.

The potential to harness the strength of different mixed-methods approaches and to combine them with ML and systematic user engagement into the MSMMF became apparent when we started researching the question: Why do some peace processes bring an end to large-scale conflict-related violence while others do not? To answer this question, we needed a methodological framework that would meet our needs to synthesize fragmented literature and evidence into sound hypotheses, to systematically collect qualitative and quantitative empirical evidence and to test and explore the hypotheses rigorously. Despite their enormous value, no existing methodological framework responded to all these needs simultaneously. While arising in the context of our own specific research project, our needs are not unique, and resonate with other scholars working on complex research puzzles, including, for example, democratic backsliding and great power politics.

In this article, we aim to make the MSMMF more widely accessible, to contribute to the advancement of research methodologies, and to offer a new resource to researchers seeking a comprehensive and adaptable approach for their own questions. We proceed as follows. First, we map the field of civil war recurrence, explaining our interest in exploring the question: Why do some peace processes bring an end to large-scale conflict-related violence while others do not? Second, we identify a methodological need to combine hypothesis development and hypothesis testing. Third, we present the MSMMF. Fourth, we illustrate its application and show that through the MSMMF we were able to consider as many explanations as possible for civil war recurrence and to identify evidence that United Nations (UN)-led peace processes are more likely to end widespread conflict-related violence, especially when peace accords also map provisions for the inclusion of women in post-conflict societies. In the conclusion, we present some limitations of the research design and reiterate its added value for future applications.

Why do some peace processes bring an end to large-scale conflict-related violence while others do not? Identifying a methodological need

The field of civil war recurrence has been growing rapidly. Most of the existing research focuses either on a few case studies (e.g., Call, 2012) or on a comparative investigation of highly specific variables among the wide range of potential factors explaining civil war recurrence (e.g., Bara et al., 2021; Loyle and Appel, 2017; Pushkina et al., 2022; Ta-Johnson et al., 2022). This generates a wealth of data and a variety of (often conflicting) findings that do not comprehensively explain why some conflicts relapse into large-scale violence and others do not.

Three partly contradictory arguments emerge from the existing literature. Some scholars suggest that civil wars resume after peace accords primarily because of contextual factors. For example, a considerable body of research suggests that adverse economic conditions lead to a resumption of violence (e.g., Collier et al., 2008; Walter, 2004). Others highlight the characteristics of the previous conflict as reasons for civil war recurrence (e.g., Nilsson and Svensson, 2021). A third group of scholars explain civil war recurrence or non-recurrence focusing on peace accords, either examining how agreements are achieved or emphasizing what provisions they contain. For example, Quinn et al. (2013) find that third-party mediation has little long-term effect on the sustainability of peace, while Gurses et al. (2008) argue that mediation increases the prospects of longer-lasting peace. In terms of peace agreement provisions, the key debates concern the impact of provisions for power sharing (e.g., Hartzell and Hoddie, 2015; Horowitz, 2014; McGarry and O’Leary, 2004), territorial self-governance (e.g., Hale, 2004; Neudorfer et al., 2022; Weller, 2009) and transitional justice (e.g., Druckman and Wagner, 2019; Duursma, 2020; Leib, 2022).

As a whole, this growing body of literature presents widely divergent findings. While highly valuable for identifying the possible factors explaining the recurrence and non-recurrence of civil war, existing publications do not capture the overall picture of the complex settings of civil wars, or the impact of the interaction of multiple factors on the likelihood of civil war recurrence. As a result, a fundamental question remained unanswered: Why do some peace processes bring an end to large-scale conflict-related violence while others do not?

We argue that this divergence partly exists because some studies focus on specific selections of explanatory factors, some focus on a small number of highly specific case studies, others rely on fundamentally different datasets and/or sources and yet others – perhaps most importantly – rely on single case-based or variance-based methodologies. Case-based approaches effectively tell us how a causal mechanism works in a specific case, but their findings are not necessarily generalizable (Beach, 2020). Conversely, variance-based approaches can test hypotheses across cases, but cannot tell us what causal mechanism is at play in a specific context (Beach, 2020). Existing mixed-methods research designs combine different research methods, trying to compensate for their respective weaknesses (Beach, 2020). However, as Figure 1 shows, they typically suggest starting either with case-based (qualitative) approaches (Beach, 2020; Goertz, 2017) or with variance-based (quantitative) approaches (Lieberman, 2005; Seawright, 2016). As a result, they prioritize either variance-based or case-based designs, without bridging the fundamental ontological and epistemological differences between these approaches (Beach, 2020). This makes ‘true multi-method research very difficult’ (Beach, 2020: 164).

Figure 1.

Multi-Stage Mixed-Methods Framework at the intersection of different mixed-method approaches.

To address this methodological need, we built on existing mixed-method research designs to create the MSMMF, a novel framework that enables the combination of the two main stages of social science research: hypothesis development; and hypothesis testing. We do not want to eliminate the methodological and epistemological divisions which are a valuable part and parcel of the social sciences. However, we hope that our efforts will enable significant advancements in fields where existing theories and evidence are fragmented and contradictory (such as civil war research), and to provide accurate policy recommendations to interested policymakers and practitioners.

The MSMMF: Purpose and design

Figure 1 compares the key features of our novel MSMMF with those of existing variance-based and case-based research designs, as summarized by Beach (2020). The diagram captures the variance in type of research questions they can answer, their strengths and their weaknesses. We build on Beach’s (2020) assertion that a combination of both approaches can ‘supplement each other’s weaknesses because they ask different questions. . . causal claims are therefore strengthened when we have evidence of both “what is the causal effect” and “how does it work here”’ (p. 165). Therefore, we intentionally developed the MSMMF at the intersection of variance-based and case-based research designs.

To ensure a genuine combination of bottom-up (qualitative) and top-down (quantitative) approaches, the MSMMF proceeds in two sequential but iterative stages, visually summarized in Figure 2: a hypothesis development stage (Stage I, Steps 1–5); and a hypothesis testing stage (Stage II, Steps 6–7). In Figure 2, the text describes all the detailed steps for the MSMMF, identifying the different methods that researchers may choose to employ at each stage of research, while the arrows capture the adaptive and iterative character of the research framework, with multiple in-built feedback loops.

Figure 2.

A step-by-step guide to the Multi-Stage Mixed-Methods Framework.

Stage I aims to develop robust, empirically grounded hypotheses. Steps 1–3 are essential to evaluate the suitability of the MSMMF for each specific research question. These steps are traditionally employed to map existing findings and identify generalizable patterns that can be formulated into hypotheses in the civil war literature. Logical thinking (Knopf, 2006) or objective thinking (Härpfer and Kaden, 2020) informed by reading is often used by positivist scholars. An alternative approach relies on systematic reviews (King, Keohane and Verba, 1994), mapping broad themes and categories of explanations, identifying gaps and formulating hypotheses informed by previous research findings. Literature reviews are typically employed by both qualitative and quantitative scholars. Qualitative researchers also employ case studies to generate hypotheses through in-depth investigations of a specific phenomenon in one or several contexts (Kapiszewski, MacLean and Read, 2015). However, some puzzles are characterized by such a wealth of valuable but conflicting findings that it becomes very challenging to formulate robust hypotheses through these methods. We recommend employing the MSMMF for this kind of complex research questions.

Steps 4–7 describe the application of the MSMMF. Given the space limitations of a single journal article, we can only briefly map the four steps necessary to apply the MSMMF. In our application section below, we briefly describe the methods we chose to employ in our research project, with dedicated boxes focusing on practitioner engagement (Box 1); machine learning (Box 2); regression and survival analysis (Box 3); and congruence analysis and process-tracing (Box 4). In our application we did not carry out an original qualitative comparative analysis (QCA) study, therefore, we have not included a dedicated box on QCA. The boxes aim to summarize a method’s objectives, applications and suggestions for further reading, but are not exhaustive explanations of the individual methods (and combinations of methods) available to researchers at each step of the MSMMF.

As Figure 2 shows, researchers can employ a combination or a selection of qualitative and/or quantitative techniques at all stages of research. This flexibility aims to enhance researchers’ agency in choosing the approach(es) for each stage of research and how to sequence and combine multiple research methods. These techniques include ML (computational social science, aimed at pattern recognition), pilot case studies (qualitative research to identify potential causal mechanisms), regression analysis (traditional statistics aimed at pattern testing), QCA (aimed at pattern recognition) and congruence analysis and process-tracing (to explore and illustrate causal mechanisms).

The research question and available data will largely guide which methods are most appropriate for each individual research project. For example, to develop hypotheses (Step 5), ML is particularly suitable for datasets with low numbers of observations and high number of variables (generally, if the number of observations divided by 30 is smaller than the number of explanatory variables). If both the number of observations and the number of variables is low, QCA is particularly suitable for hypothesis development. Hypothesis development through pilot case studies may be more or less feasible depending on highly volatile context-specific socio-political conditions, especially in conflict settings. For all methods applied in the MSMMF, we trust that researchers will adhere to common practices in political science, including considerations of significance levels, model fit, truth tables and ethical research practices (see also the methods boxes for further details).

Due to the application of multiple methods concurrently, Step 5 of the MSMMF can result in three possible outcomes. Ideally, all the methods employed will identify similar or compatible hypotheses, allowing researchers to retain and test them in the next stage. Alternatively, different methods might identify diverging hypotheses. Such diversity is not problematic at the hypothesis development stage, so we recommend retaining all hypotheses for testing in Stage II of the MSMMF. Finally, the methods employed at Stage I may lead to extremely contradictory hypotheses. In this case, we recommend retaining all of these – however contradictory – hypotheses for hypothesis testing, as even less robust findings at Stage I will ultimately strengthen the rigour and explanatory power of the research at Stage II.

Step 6 in Figure 2 lists the methods that can be combined or selected to test and explore the hypotheses. For Stage II, we strongly suggest employing both qualitative and quantitative research methods in parallel to test hypotheses as thoroughly as possible. In so doing, researchers will maximize the benefits of the MSMMF, bridging more effectively the qualitative–quantitative divide. The choice of specific techniques will depend on the research question and available data.

Finally, Step 7 brings together the evidence to formulate overarching conclusions. Moran-Ellis et al. (2006) provide a useful illustration of different approaches to combine and synthesize findings that may guide researchers at this stage. Depending on individual preferences and research questions, integration of methods, integration of analysis and/or integration of theories may be appropriate to individual research projects (Moran-Ellis et al., 2006: 47–49). We recommend, at this stage, to thoroughly triangulate findings from the multiple research methods employed at Step 6 through the consultation with practitioners during dedicated user workshops.

The MSMMF embeds consultations with practitioners throughout the research process. These are signposted in Figure 2 as ‘user workshops.’ While we label these consultations as ‘workshops,’ their specific format may vary from project to project, and may encompass surveys, interviews, informal communications, etc. (see Box 1). In our case, alongside a series of bespoke workshops, we created a dedicated advisory board, and consulted the members at pivotal stages of research when decisions had to be made on case selection, hypothesis development, testing and dissemination of findings. Our engagement activities focused on individuals interested in employing evidence-based practices for peacebuilding, mediation and conflict prevention, but the population of users might vary depending on the specific research question. We identified interested practitioners through our professional networks, the existing literature and the text of peace agreements. We selected them based on their expertise and potential insights, but also ensured a diverse participation in our events to give voice to different perspectives (geographical, professional and gender). In total, we consulted approximately 100 practitioners, including officials from governments and international organizations, mediators, personnel working in one of the 11 countries in the Conditions of Recurrence Dataset (CoR-D, see also Figure 3R), practitioners in non-governmental organizations, civil society actors and academics. Our project had full ethical approval from our university’s research governance committee, and all participants consented for discussions to be included in our outputs without attributing individual remarks (Chatham House Rule). These sessions provided opportunities to transfer knowledge between academics and practitioners through an iterative and long-term process of repeated engagements, and ensured the relevance and visibility of our project.

Box 1. Practitioner and user engagement.

Encompassing a variety of activities to engage research users at different stages of research projects, including presentations, workshops, consultancy, secondment, briefings and dialogue, commissioning of research, submission of evidence, educational content, games and simulations (Bobo et al., 2024).

Aims: ensuring relevance of research and findings; expose academics to additional puzzles; sense-check findings; and collaboratively address complex real-world problems (McCabe et al., 2023).

Use: formal and informal engagement including through presentations, workshops, consultancy, secondment, briefings and dialogue, commissioning of research, submission of evidence, educational content, games and simulations (Bobo et al., 2024).

When to involve practitioners: depending on the policy relevance of a question.

Further reading and applications: McCabe et al. (2023); Bobo et al. (2024).

Figure 3.

Universe of cases of the Conditions of Recurrence Dataset (for hypothesis development).

Application and findings

Our research project on civil war recurrence provided a real-world context for the development and application of MSMMF. In Step 1, we formulated our research question: Why do some peace processes bring an end to large-scale conflict-related violence while others do not?

We then carried out a review of the existing literature to identify the different arguments and explanatory factors considered in previous studies, as well as potential gaps (Step 2).

In Step 3, we identified our universe of cases, and carried out additional desk-based research to identify case-specific explanatory factors for the recurrence and non-recurrence of conflict. We set out to map all the past cases of peace processes that experienced conflict recurrence before achieving the termination of conflict-related violence between the same conflict parties through a peace accord between 1975 and 2015. Our underlying reasoning was that by systematically examining factors of recurrence and non-recurrence within the same peace process we could identify more robustly the factors facilitating stable peace. We identified 14 peace processes in 11 countries (highlighted in Figure 3): the Aceh Conflict; the Angolan Civil War; four separate but related conflicts in Bangsamoro (Bangsamoro I with the Moro National Liberation Front; Bangsamoro II with the National Democratic Front; Bangsamoro III with the Cordillera People’s Liberation Army; and Bangsamoro IV with the Moro Islamic Liberation Front); the Bougainville Conflict; the Burundian Civil War; the First Ivorian Civil War; the Liberian Civil War; the Malagasy Political Crisis; the Mali Conflict; the Myanmar Conflict; and the Sierra Leone Civil War. These 14 peace processes represent the full universe of peace processes that experienced conflict recurrence before achieving the termination of conflict-related violence between the same conflict parties through a peace accord between 1975 and 2015.

One of the key innovations of the MSMMF is the systematic and repeated engagement with users and practitioners at all stages of research. Through a first user workshop, we consulted with practitioners to validate or challenge case selection and to identify further explanatory factors and mechanisms that may be missing in the literature but may seem ‘obvious’ to them. This further step increased the robustness of our findings and their relevance for potential users. Both in this workshop and in following consultations, we occasionally gathered different and contrasting opinions from the practitioners taking part in the events. Where these differences appeared irreconcilable, we employed opinion polls in Zoom to accurately record opinions, and then used them to inform our team’s collective decisions.

In Step 4, we explored existing datasets to evaluate if they were suitable to empirically investigate our research question (i.e., if they captured enough explanatory factors for civil war recurrence and non-recurrence). Because no sufficiently fine-grained dataset existed, we created a dataset including the factors and outcomes of interest identified through the literature review and user workshop: the Conditions of Recurrence Dataset (CoR-D). To build the CoR-D, we downloaded the full text of all the agreements comprised in the 14 peace processes from the UN Peacemaker website (a total of 159 peace agreements, or observations). We coded all the agreements manually using NVivo to identify features of the settlement and of the negotiations. To capture conflict-related and contextual factors, we drew on existing datasets such as Ethnic Power Relations (Vogt et al., 2015), Quality of Government (Teorell et al., 2021) and the Uppsala Conflict Data Program (Gleditsch et al., 2002; Pettersson, 2020; Pettersson et al., 2021). As a result, the CoR-D captures 235 fine-grained variables (59 contextual and conflict-related factors, and 177 peace agreement provisions) for 159 observations (peace accords) in 14 peace processes.

For other research questions, where relevant large datasets already exist, it may be possible to employ an existing dataset for hypothesis development and testing, with three main stipulations. First, the dataset needs to be adaptable for ML, regression-based analysis (inferential statistics) and/or QCA, depending on the research question and on researchers’ methodological preference. The choice of method(s) in Step 5 of the MSMMF is not predetermined, but depends on the research question, on existing datasets and/or the availability of resources to build one. Different estimation methods might require slightly different datasets: QCA is better suited for datasets with a limited number of observations and variables (Marx, 2006); inferential statistics requires a random sample (Agresti, 2018); and ML needs a dataset large enough to be split into training and testing sets (James et al., 2013, see also Figure 4).

Figure 4.

How to split existing datasets for the Multi-Stage Mixed-Methods Framework.

Second, a dataset (or subsection of a dataset) needs to be employed for the hypothesis development stage (DSg in Figure 2) and a different unseen dataset (or subset of a dataset) needs to be used for the hypothesis testing stage (DSt in Figure 2). As Figure 4 suggests, when splitting an existing dataset into two subsets, we recommend a random 50/50 split for developing hypotheses (DSg) and hypothesis testing (DSt), ensuring an appropriate balance in the outcomes. A 50/50 split helps to prevent overfitting during hypothesis development and ensures a sufficiently large dataset for testing hypotheses. Alternative split practices could be 70/30 and 80/20 splits as it is commonly done in the field of artificial intelligence. However, we contend that it is vital to ensure that researchers have a substantial proportion of their data reserved for testing of their hypotheses which would allow for robustness tests to be conducted with ease and enhance the rigour of research.

Third, when employing supervised ML at Stage I of the MSMMF, DSg should be further split into a training and testing dataset as per best practice in ML. The sequential splits are visually represented in Figure 4.

In Step 5 of the MSMMF (hypothesis development), we employed ML, alongside pilot case studies, the findings of the existing QCA-based literature and practitioner consultations. Due to its ability to shed light on patterns explaining the outcome (as summarized in Box 2), supervised ML is useful to identify variables for hypothesis development (Step 5A of the MSMMF). We suggest employing it in combination with in-depth case studies (Step 5C) and/or QCA (Step 5B) to develop hypotheses.

Box 2. Machine learning (ML).

Encompassing lasso and sparse regression, classification and regression trees, boosting and support vector machines. In supervised ML the dependent variable has already been labelled by humans. Unsupervised ML automatically groups observations without human intervention (James et al., 2013).

Aims: in political science, ML is primarily used for prediction, often achieving significantly higher accuracy than regression models. This is partly because whereas regression models perform well in deductive hypothesis testing, regression is not used for inductive learning (Cranmer, 2019). Embedding ML in the Multi-Stage Mixed-Methods Framework can help advance research puzzles characterized by a small number of observations/cases and a high number of explanatory variables.

Use: ML involves training machines to enhance their performance on specific tasks using algorithms and statistical models, without explicit programming for those tasks. The data used for learning are termed the training set, while the data for applying learned knowledge are the test set, with no overlap between the two.

Sample size and number of variables: no upper or lower boundaries of how many variables per observation are needed.

Further reading and applications: James et al. (2013); Cranmer (2019); (Desmarais and Cranmer, 2013; Klarner, 2018; Ward et al., 2010 – as referenced by [Cranmer, 2019: 3]).

We carried out supervised ML in the form of decision trees (classification and regression trees or Classification and Regression Trees Analysis (CART), James et al., 2013) to identify factors consistently associated with our outcome of interest (the non-recurrence of civil war). Figure 5 summarizes the steps involved in applying CART in our own research. As a method, CART offers the advantage of being easily interpretable, making it accessible to both academics and practitioners without extensive statistical training. CART operates through recursive partitioning, a process that repeatedly divides data into groups that are as homogeneous as possible. In so doing, the algorithm identifies variables that best predict an outcome. The quality of the factors identified in the training dataset is then evaluated by using these factors to predict the dependent variable in the testing dataset. Good training models will have a high percentage of correctly predicted observations in the testing dataset. This procedure might be familiar to scholars working with maximum likelihood regression analysis, where the fit of an inferential statistical model is tested by examining the correctly predicted observations.

Figure 5.

How to carry out Classification and Regression Trees Analysis.

In our case, CART identified UN leadership, and peace agreement provisions for the inclusion of women in post-conflict societies and for plural justice mechanisms as accurately predicting non-recurrence of civil war in the training dataset. These three factors also had a high predictive accuracy when employed to predict non-recurrence in the testing dataset. Based on supervised ML, we therefore identified three possible explanatory variables which appeared associated with the end of civil war in peace processes that experienced prior conflict recurrence: UN leadership of the mediation process; provisions to include women in post-conflict societies; and provisions for plural justice (visually represented in Figure 6).

Figure 6.

Results of our Classification and Regression Trees Analysis.

Because of the COVID-19 pandemic, we could not travel and carry out fieldwork to develop hypotheses concurrently with supervised ML. Instead, we conducted remote expert interviews and online focus group discussions on all of the 14 peace processes in CoR-D. We also did not carry out an original QCA study, but instead we cross-referenced our CART findings with two recent QCA-based studies on the characteristics of resilient peace accords: Fontana et al. (2021b); and Pushkina et al. (2022). Our CART results were broadly consistent with their emphasis on the beneficial impact of third-party involvement in peace processes, and of the inclusion of formerly marginalized groups in peace accords. Finally, we convened a second consultation with practitioners, which helped us formulate and refine hypotheses on specific causal mechanisms, and to identify suitable case studies for subsequent hypothesis testing.

In sum, in Stage I of the MSMMF, and especially in Steps 4–5, we developed hypotheses on the factors associated with the end of recurrent civil war: the CART suggested that UN-led mediation, provisions to include women in post-conflict societies, and plural justice provisions are associated with the end of recurrent civil war. However, these insights are not generalizable beyond the 14 peace processes in CoR-D and provide few insights on a specific causal mechanism, that is, how UN leadership, provisions for plural justice and provisions to include women in post-conflict societies may help to end cycles of recurrent civil war.

Stage II of the MSMMF (Steps 6–7) allows researchers to test and explore the hypotheses formulated in Stage I, and particularly to establish whether and how the identified factors have a robust and significant causal relationship with the outcome of interest. In our case, we explored whether UN leadership, provisions for plural justice and provisions to include women in post-conflict societies have a robust and significant causal relationship with the end of conflict-related violence beyond the 14 peace processes in CoR-D, and how these factors may help prevent the resumption of violence after a peace accord.

At Stage II of the MSMMF, we recommend testing and exploring hypotheses in parallel through quantitative and qualitative methods (Step 6) to maximize insights and minimize unintended biases. We recommend carrying out quantitative hypothesis testing (Step 6A) through inferential statistical analysis of an unseen dataset (also see Box 3). As summarized in Figure 4, existing datasets should be adapted to employ the (unseen) part of a larger existing dataset at this stage. Different regression methods may be appropriate to different questions and datasets (including ordinary least squares or maximum likelihood estimations, cross-country or time-series cross-sectional analysis).

Box 3. Regression and survival analysis.

Aims: regression analysis allows researcher to investigate the strength of an association between independent and dependent variable(s) to test pre-formulated hypotheses.

Survival analysis is used to estimate the time until an event – such as the recurrence of war – occurs. Cox survival analysis makes no assumptions about the hazard over time. This means that the hazard of failure could be increasing and then decreasing, or decreasing and then increasing, or remaining constant over time.

Use: as described by Agresti (2018), they employ a random sample of the population to make inferences about the population (inferential statistical analysis).

Regression analysis deals with continuous or categorical outcomes and is therefore appropriate to answer questions such as ‘will fighting resume after a peace accord?’ With a binary dependent variable, it is advisable to use maximum likelihood estimation. For interval-level variables, it is advisable to apply ordinary least squares regression.

Survival analysis deals with time-to-event data and is typically used for questions such as ‘how long does peace last?’.

Sample size and number of variables: it is recommended to have at least 30 observations per variable.

Further reading and applications: Agresti (2018); Wooldridge (2015); Hartzell and Hoddie (2007)

In our case, we used a separate dataset for hypothesis testing due to the highly specific nature of CoR-D (which encompasses the whole universe of cases). This enabled us to accommodate the strong demand from practitioners to be able to test our findings on more cases. We selected the dataset of Political Agreements in Internal Conflicts (PAIC) (Fontana et al., 2021a) which best fulfilled our needs with respect to country coverage, time frame and existing control variables, but also captured more peace processes than CoR-D, including situations where negotiated settlements immediately led to the cessation of extensive violence. Following existing studies on peace agreements (Hartzell and Hoddie, 2007), we tested our hypotheses through Cox proportional hazard regression analysis. Our analysis confirmed that UN leadership and provisions for the inclusion of women in post-conflict society are associated with the end of civil war globally. Conversely, provisions for plural justice have no robust and significant relationship with the end of conflict-related violence beyond the 14 peace processes in CoR-D. To test robustness, we repeated the analysis by only including accords that experienced at least one previous relapse into conflict. These observations encompassed agreements addressing recurrent conflicts where no stable settlement was achieved, agreements concluded after 2015 (such as Colombia’s Acuerdo Final), as well as those agreements in CoR-D that were concluded in the timeframe considered by the PAIC dataset (1989–2016). The results remain robust.

In parallel with quantitative hypothesis testing, we engaged in qualitative research on selected case studies (Step 6B). As Figure 2 shows, we recommend employing process-tracing or congruence analysis to explore how the variables identified in Stage I may cause the outcome of interest (Beach and Pedersen, 2013). As Box 4 summarizes, process-tracing allows scholars to identify and map complex causal mechanisms and draw within-case inferences (Beach and Pedersen, 2013, 2019). However, it presupposes a linear and unbroken chain of ‘action and reaction [. . .] that connects the potential cause with its hypothesized outcome’ (Wauters and Beach, 2018: 288). Congruence analysis is better suited to explore and illustrate possible non-linear explanations between the explanatory factors and outcome of interest (Wauters and Beach, 2018). In our study, we opted for congruence analysis to explore the relationship between the end of civil war and UN leadership, provisions for plural justice and provisions to include women in post-conflict societies.

Box 4. Congruence analysis and process-tracing.

Aims: congruence analysis and process-tracing employ within-case inference to explore and explain the relationship between explanatory factors and an outcome of interest.

Use: congruence analysis investigates correlations between explanatory factors and an outcome of interest.

Process-tracing investigates the workings of the mechanism (s) that contribute to producing the outcome of interest by tracing the theoretical causal mechanism(s) linking explanatory factors and an outcome of interest. They are typically presented as ‘a stepwise test of each part of a causal mechanism, especially in the theory-testing variant’ (Beach and Pedersen, 2013: 5–6).

Selection of cases: depending on the research question, it may be appropriate to choose typical cases, deviant cases, or both. Typical cases (representative of the population) would exhibit both the explanatory factors and outcome of interest (Goertz, 2017: 247). Deviant case studies would only exhibit the explanatory factors or outcome of interest. Case studies can be identified based on the size of their residuals in relation to a regression-based analysis of X:Y correlations (Lieberman, 2005: 444, see also Gerring and Seawright, 2006: 89). These guidelines are particularly relevant when investigating the magnitude of causal effects (Beach and Pedersen, 2018; Gerring, 2007; King et al., 1994; Lieberman, 2005; Tarrow, 2010).

Further reading and applications: Beach and Pedersen (2013, 2019); George and Bennett (2005); Tannenwald (1999).

To identify suitable case studies, we ran a standard multivariate regression analysis of CoR-D, including the factors identified in our hypothesis development (Step 5A). We included UN mediation, provisions for plural justice and provisions to include women in post-conflict societies (alongside a range of standard control variables) as explanatory variables, while the outcome was the end of widespread conflict-related violence. This analysis identified both typical case studies (exhibiting explanatory variable and outcome of interest) and deviant case studies (exhibiting explanatory variable or outcome of interest). We suggest that depending on the specific research question, it may be appropriate to choose typical cases, deviant cases, or both. We were interested in how specific provisions may explain the end of recurrent civil war, so we followed Goertz in choosing a ‘case study [to] be representative of the population or causal effect. The case studies should explore a typical case of the causal effect, where “typical” is a synonym for “representative”’ (Goertz 2017: 247). To isolate the most representative cases among cases where both our explanatory variable and our outcome of interest were present (typical), we also looked at the change in predicted probability of conflict recurrence after the introduction of our hypothesized explanatory factors. After selecting cases that had high changes in probability, we engaged in a third set of practitioner consultations to explore and evaluate the suitability of each possible case. This process resulted in our selection of three peace processes for further qualitative analysis: Sierra Leone; Liberia; and Bangsamoro (the Philippines).

We subsequently carried out congruence analysis to determine how the end of Sierra Leone’s recurrent civil war was linked with UN leadership; how the end of violence in Liberia was linked with provisions for the inclusion of women in post-conflict societies; and how the end of conflict in Bangsamoro was linked with provisions for plural justice. This relied on in-depth case studies, analysis of documentary material and official reports from governments, international organizations and local and international non-governmental organizations, and interviews with mediators, practitioners and experts who worked in and on the three locations. The mechanisms linking plural justice provisions with the end of violence in the Philippines were ambiguous and case-specific (Deinla, 2019). However, research on Sierra Leone and Liberia led to more generalizable findings on the beneficial impact of UN leadership in peace processes, especially when combined with the inclusion of women in post-conflict societies. In a fourth set of consultations with practitioners, we returned to our potential users to evaluate the plausibility of our causal mechanisms and reflect on how to best disseminate our findings.

By applying the MSMMF to answer the question Why do some peace processes bring an end to large-scale conflict-related violence while others do not? we found that the combination of UN leadership of the mediation process and provisions for the inclusion of women in post-conflict societies helps to build, legitimize and empower broad coalitions of local, national and international actors committed to the negotiation, implementation and operationalisation of peace settlements, contributing to preventing civil war recurrence. This not only applies to recurrent civil wars: our wider testing suggests that these two factors can stop the vicious cycle of repeated civil war recurrence before it even starts.

Conclusion

In this article, we outlined a new methodological framework to combine hypothesis development and hypothesis testing into a coherent and robust multi-method research design: the MSMMF. The MSMMF contributes to methodological advancement in the social sciences in three respects. First, it combines the strengths of existing mixed-method approaches in a novel way, providing researchers with a rigorous framework to formulate theoretically sound hypotheses and derive empirically grounded findings. This enables researchers to tackle extant questions in fragmented and contradictory research fields. Second, the MSMMF pilots the use of supervised ML (computational social science) for hypothesis development, taking multi-methods designs into the machine-learning age. Third, the MSMMF embeds iterative opportunities for engagement with practitioners at all stages of the research process. This enables researchers to address questions of high policy relevance while ensuring that the insights of practitioners inform research (and vice versa), bringing researchers and practitioners together to find robust, evidence-based solutions to existing global challenges.

Despite the benefits of the MSMMF, there are some limitations to the framework and its applicability. Due to the variety of methods employed, the MSMMF is best suited to large research teams with diverse methodological expertise. While collaborative work has become increasingly common, the challenge of working across methodological cultures is well documented, so the successful application of the MSMMF requires cooperative and mutually supportive research teams. Alternatively, the MSMMF is suited to a researcher able to acquire an eclectic methodological expertise: methods training is a standard component of graduate programmes and additional expertise can be acquired through a variety of methods schools.

The MSMMF can also appear resource-intensive and time-intensive. However, its costs may be minimized by employing pre-existing datasets rather than generating new ones, where suitable ones are available. For example, there is a wealth of datasets on territorial self-governance (Neudorfer et al. 2025), and a lively but inconclusive debate on the impact of federalism, decentralization and autonomy on civil war occurrence and duration (Fontana et al., 2021a). The MSMMF could be employed to synthesize all existing findings and data into robust hypotheses, and then testing and exploring them systematically to advance the debate.

Finally, the MSMMF is not a silver bullet for tackling and answering all research puzzles. Social science is continuously evolving, and we see the MSMMF as contributing to this wider evolution by enabling a more joined-up, comprehensive research process. No methodological framework, including the MSMMF, should or could claim to solve all research questions. However, the MSMMF uniquely combines the strengths of case-based and variance-based mixed-methods approaches, embeds supervised ML and maps engagement with practitioners at all stages of research. Its added value is most apparent when tackling contradictory and fragmented research fields with a high policy relevance. The MSMMF may be less suited for more specific research questions that address the relationship between one specific factor and one specific outcome of interest, where researchers may use other existing research designs. Our application, however, shows that for research questions focusing on causes-of-effects, where there is a multiplicity of contrasting theories, data and empirical findings, the benefits of the MSMMF outweigh its costs.

In this paper we summarized how – through the MSMMF – we explored an important but still not conclusively answered question: Why do some peace processes bring an end to large-scale conflict-related violence while others do not? The MSMMF enabled us to examine the widest possible range of explanatory factors, explore holistically the peace processes tackling recurrent civil wars, and test and explore our hypotheses across a wider global set of peace agreements. By employing computational, quantitative and qualitative methods throughout our research, we were able to show that UN leadership of the mediation process combined with provisions for the inclusion of women in post-conflict societies, can help to prevent the recurrence of civil wars by creating broad coalitions of actors committed to negotiating, implementing and operating a peace settlement. These findings can inform future negotiations and policies, and they would not have been possible without the MSMMF’s systematic combination of supervised machine learning, in-depth case studies, congruence analysis, inferential statistical analysis and practitioner engagement.