Sage Journals: Discover world-class research

Abstract

As more researchers have considered the use of mixed methods, writings have moved away from debates about epistemological incompatibilities and now focus on the (potential) value of increased understanding that comes from combining qualitative and quantitative approaches. Yet, as the level of integration can vary substantially, some designs are said to allow one method or the other to dominate. Although there may be sound reasoning for intentionally allowing one method to dominate, here we investigate one literature as a moment to reflect why, and on the degree to which mixed methods sequence is so bound up with methodological dominance, that calling such studies “mixed” may seem misleading. Like the history of social science more generally, it is quantitative research that is typically given more weight in these studies and academics have noted a few reasons why this may be the case. Few have investigated how research design—and more specifically method sequence—may impact method dominance. Using an emerging mixed methods literature surrounding the social acceptance of wind energy (N = 34), we study the relationship between the timing of each method (i.e., sequence) and method dominance to see whether qualitative methods in particular are marginalized. Through our Dominance in Mixed Methods Assessment model, we provide evidence that indeed qualitative methods are marginalized and this may be associated with method sequence and other design elements. Moreover, some authors focus solely on one method, giving pause to caution both writers and readers about the use of the term “mixed methods.” The analytical approach is detailed enough to be replicated and detect whether these patterns are repeated in other research domains.

Keywords

mixed methods qualitative dominance sequence research design wind energy DIMMA

Introduction

The rapid growth of mixed methods research has reinvigorated discussions surrounding why (and how) mixing quantitative and qualitative approaches should be done. Debates started in the mid-19th century and focused on the tensions between stand-alone quantitative and qualitative approaches (see Becker & Geer, 1957; Trow, 1957). Today, mixed methods research has created a “booming field of methodological and theoretical discussions” (Flick, 2017, p. 46) surrounding the qualitative/quantitative dichotomy (Glassner & Moreno, 2013), difficulties in publishing (Mertens, 2011), and method integration (Mason, 2006; Mertens, 2014; Sligo, Nairn, & McGee, 2017). Despite this increased popularity, there is a relative lack of research critically looking at the underpinnings of how mixed methods are practiced and published (S. Hesse-Biber, 2010a; Heyvaert, Hannes, Maes, & Onghena, 2013; Hong & Pluye, 2018; Patry, 2013).

Through an examination of how mixed methods are typically understood, Creswell and Plano Clark (2018) write about approaches to classifying mixed methods research design (see also the “five major questions” from Bryman, 2006). They suggest there are four major features that help us understand the decisions and characteristics of mixed methods: purpose (or intent) for mixing, sequencing of qualitative and quantitative strands, priority (dominance) of each method, and level of interaction between each strand. This article concentrates on the “two main factors” of sequence and dominance (Molina-Azorín & Lopez-Gamero, 2016; Morse, 1991; Morgan, 1998). Sequence relates to questions of method order, the most basic being whether methods are implemented simultaneously or one after the other (Morgan, 2013). Dominance relates to emphasis or which method is more central to the paper (Creswell & Plano Clark, 2011). While both of these ideas have received significant consideration in the literature, we add how the two may be interrelated—an idea that has received little attention. We do so through the analysis of a single literature—the social acceptance of wind energy. In doing so, we use the literature as a case study—creating a moment to ponder the broader implications of sequence. More specifically, we investigate whether dominance is more likely when qualitative methods are deployed at the beginning, end, or highly integrated with quantitative methods (Bryman, 2006; Denzin, 2010).

Dominance in Mixed Methods Research

Dominance has been a central point of conversation in the study of mixed methods research. The term is somewhat synonymous with the other terms including priority, weighting, emphasis, and status and will be considered as such for the purposes of the analysis here. Others have preferred to think of this idea in terms of qualitative-driven and quantitative-driven research (S. Hesse-Biber, 2010b; S. N. Hesse-Biber, Rodriguez & Frost, 2015; Mason, 2006). Although methodological balance is not a requirement of mixed methods per se, there is value what Creamer (2018) calls meaningful interaction and representation—especially when researchers use the term “mixed methods.” At the very least, studying dominance may provide a useful point of reflection, since there are various processes (e.g., team research culture, editorial procedures) which may lead to unanticipated emphasis of one method or the other.

Much of the discussion on method dominance has originated from the concern that qualitative methods are typically marginalized in mixed methods research (Bryman, 2007; Giddings, 2006; J. C. Greene, Caracelli, & Graham, 1989; Niglas, 2004). S. Hesse-Biber (2010b) has highlighted the methodological orthodoxy whereby qualitative research serves mostly secondary roles. She states there is a fear among researchers that conducting mixed methods research in this way results in “an ‘adding and stirring’ of qualitative methods that often takes the form of sprinkling in some vignettes to provide narrative examples of the conclusions already reached by means of quantitative methods” (p. 457). Others from the qualitative research community in particular, see the quantitative as subordinating “tokenistic” (Ivankova & Plano Clark, 2018) qualitative research. This often results in “QUAL-light research that does not deliver what it has promised” (Teddlie & Tashakkori, 2012, p. 777).

Beyond these “qualitative communities,” there is fairly strong support for the idea that qualitative methods generally get subjugated in mixed methods research. Reviews of different literatures in the social sciences (Creswell, Fetters, & Ivankova, 2004; Harrison & Reilly, 2011; McManamny, Sheen, Boyd, & Jennings, 2015; O’Cathain, Murphy, & Nicholl, 2007; Plano Clark, Huddleston-Casas, Churchill, Green, & Garrett, 2008) have found that qualitative methods and/or findings are usually not given priority. Some have even defined mixed methods as research that contains one complete method alongside “one or more…supplementary components” (Morse & Niehaus, 2009, p. 9). Indeed, through a recent analysis of empirical mixed methods research, only 22% were assessed to have equal dominance (Creamer, 2018).

Of course, this trend of quantitative dominance in mixed methods is not universal. Indeed, others in this area including Creswell and Plano Clark (2011) point out that qualitative methods may well dominate in mixed methods studies (see also Creswell, Shope, Plano Clark, & Green, 2006; Mason, 2006). There is also a more recent paper from Archibald, Radil, Zhang, and Hanson (2015) which shows qualitative results were prioritized in 86% of mixed methods papers (N = 94) published from 2003 to 2014. However, the authors limited the scope of their sample to “leading qualitative journals” and thus we should not be surprised to see these methods playing more prominent roles. More broadly, discrepancies in terms of the patterns of method dominance may relate to the conventions of particular fields of study, the context of research questions, and exactly who is defining “true” mixed methods (S. Hesse-Biber, 2016; Maxwell, Chmiel, & Rogers, 2015). Seifert, Goodman, King, and Magolda (2010) offer such a rare example of a study in which neither method is dominant. The authors claim they were able to do so by (i) articulating an integrative purpose, (ii) creating equally important research questions, and (iii) having two separate research teams. The last of these efforts therefore avoided the challenge of insufficient expertise in one method (Creswell & Plano Clark, 2011).

While method dominance has long been an important object of study (Creswell and Plano Clark, 2011; Hanson, Creswell, Clark, Petska, & Creswell, 2005; Leech & Onwuegbuzie, 2009), the latest influential text by Creswell and Plano Clark (2018) suggests it is an idea that some are shifting their focus away from in favor of a study’s intent. This has been caused in part because researchers are finding it difficult to ascertain the “vague and often confusing priority” of mixed methods studies (p. 5), which is especially the case when authors do not clearly state the method priority of their research (Plano Clark et al., 2008; Schram, 2014). Still other emerging literature continues to study or discuss method dominance as an important concept shaping mixed method typologies (Gibson, 2017; Lucero et al., 2018; Shannon-Baker, 2016; Turner, Cardinal, & Burton, 2017).

Combining these ideas, we study dominance here to spur what may be a waning discussion, increase our understanding of the ways to meaningfully to look at the concept, and make it less confusing for others going forward. Emphasizing one method over the other has mostly been thought to be a conscious choice of the researcher or is the result of pragmatic conditions such as the expertise, the scope of literature grounding the study, publication timelines, or the audience for the study (Arnon & Reichel, 2009; Creswell, Plano Clark, Gutmann, & Hanson, 2003; Shannon-Baker, 2016). Here, we argue that there may be other processes at work. In this case, method sequence may also be playing a role.

Sequence in Mixed Methods Research

The question of method sequence is detailed by Morse (1991) who asks the simple question: Are the data collected simultaneously or sequentially? Although some researchers have since examined more complex issues surrounding timing in mixed methods research through systematic reviews (Cameron, 2009; Palinkas, Horwitz, Chamberlain, Hurlburt, & Landsverk, 2011) or methodological thought pieces (Guest, 2013; Leech & Onwuegbuzie, 2009), the “why” of sequence is more often the focus.

Driven by the variety of classification schemes (Hanson et al., 2005), some have suggested that there are up to 40 mixed methods designs (Ivankova, Creswell, & Stick, 2006; Plano Clark & Ivankova, 2015). Yet, for the purpose of this study, including its focus on method sequence, we narrow the scope considerably. Using a combined set of criteria outlined by Creswell and Plano Clark (2018) and shaped by Holstein (2014) and Creamer (2018), we use a four families of [mixed method] design. Although the basic tenets and most of the terminology is taken from Creswell and Plano Clark (2018), we draw upon Holstein’s (2014) categorization largely because she groups research design based on method sequence. Below, we describe the four designs (exploratory sequential; explanatory sequential, convergent, and fully integrated) which provide important reference points in our analysis.

Sequential designs

The most important features of sequential designs are the use of quantitative and qualitative methods, one after the other. Most often findings from the first method feed into the design of the second (Teddlie & Tashakkori, 2006). In some but not all cases of qualitative followed by quantitative methods, the qualitative will act mainly as a “prestudy” to the quantitative research (Glaser & Holton, 2007). Creswell and Plano Clark (2018) label this the exploratory sequential design (Holstein, 2014, reverses the order, i.e., sequential exploratory) and may be used when the important issues need to be identified or to test the validity of qualitative findings on a wider population (S. Hesse-Biber, 2010b). In this case, the qualitative will act to shape quantitative methods that follow—a particular strategy known as the instrument development design (Harrison & Reilly, 2011). Especially when qualitative methods are used in this way, the quantitative will typically dominate (Kettles, Creswell, & Zhang, 2011). In other cases, qualitative methods have been thought to dominate exploratory studies (Creswell and Plano Clark, 2011; Jeanty & Hibel, 2011; Stentz, Clark, & Matkin, 2012).

The rationale for the mixed method explanatory sequential design (or sequential explanatory design as in Holstein, 2014) is often that the quantitative “…analysis provide a general understanding…[while the] qualitative data and their analysis refine and explain those statistical results” (Ivankova et al., 2006, p. 5). Some may use survey-based methods as a “prestudy” for later research to be examined more thoroughly or to explore contradictory results through qualitative methods (S. Hesse-Biber, 2010b). Initial quantitative results may also be used to more purposefully select participants for the qualitative strand to follow (Gelo, Braakmann, & Benetka, 2008). Although this may seem to imply the dominance of qualitative methods, most of the literature still suggests quantitative methods usually take priority (Creswell and Plano Clark, 2018; Doyle, Brady, & Byrne, 2016; Harrison & Reilly, 2011; Jeanty & Hibel, 2011; Stentz et al., 2012).

Nonsequential designs

In contrast to the designs described above, the convergent design (or parallel design as in Holstein, 2014) usually deploys both methods simultaneously. Component or subresearch questions are framed from the start as opposed to sequential designs where the results of one will influence the other (Tashakkori & Creswell, 2007). Each stage of data collection and analysis need not be completed at the same time; however, they tend to be separated and the results of one usually do not affect the design of the other. While less has been written about this design, Doyle, Brady, and Byrne (2016) suggest that the approach is more likely to result in balanced qualitative and quantitative methods.

The second nonsequential design, the fully integrated design, is what Teddlie & Tashakkori (2006, p. 23) call “the Full Monty” of mixed methods research design. Creamer (2018) defines it as “an approach…where there is the intention to mix or integrate the qualitative and quantitative strands of the study throughout each of the stages or phases of the research process” (p. 12). Most often used in inductive forms of inquiry, the approach combines quantitative and qualitative methods interactively along all stages of the research process. In some cases, this will mean qualitative and quantitative methods will alternate (see J. S. Greene & Geisken, 2013) depending on the evolution of the research.

The Interactive Effects of Sequence and Dominance

Sequence and dominance have been discussed in the literature in isolation; however, less attention—especially through empirical research—has been paid to the potential interactions between them. Recently Archibald et al. (2015) suggested that research design can impact method dominance, hinting that sequence may be involved. Thus, there is a need for research aimed at increasing our understanding of the potential for method sequence to impact which method, if any, plays a dominant role.

Notwithstanding some commentary (e.g., Creswell and Plano Clark, 2011), and given the history of qualitative methods in mixed methods research, we suspected there may be subjugation (lack of dominance) of qualitative methods in particular (Bryman, 2007; S. Hesse-Biber, 2010a). This working hypothesis was also inspired by personal experiences with manuscript publication wherein we felt pressures from reviewers and/or editors to emphasize quantitative findings (see also Bryman, 2007).

The small but growing empirical literature which does address both sequence and dominance (see Ivankova et al., 2006; McManamny et al., 2015; Plano Clark et al., 2008; Stentz et al., 2012; van der Roest, Spaaij, & van Bottenburg, 2015; Žydžiumaite, 2007) generally analyzes the two dimensions separately, further suggesting that method dominance is a choice that researchers make separate from other methodological considerations. In their research studying mixed methods research published in Business Strategy and the Environment, Molina-Azorín and Lopez-Gamero (2016) do look at method sequence and dominance but do not fully examine how the former may influence the latter. Others including Leech and Onweuegbuzie (2009) have only depicted mixed method research as either being done sequentially or simultaneously and do not consider how method sequence may influence how research is conducted or received. Palinkas, Horwitz, Chamberlain, Hurlburt, and Landsverk (2011) have moved past this in describing research that uses qualitative methods to inform quantitative surveys, for example, but their discussion does not go far beyond that. Instead, they justify their findings—which highlight the prevalence of exploratory sequential approaches in health-care research—by stating the “focus is…consistent with [calls] by funding agencies” (p. 48). Even within some methodologically targeted journals, research has largely ignored the possible interactive effects of method sequence and method dominance (e.g., Hall & Howard, 2008).

Through their recent work outlining and assessing mixed methods phenomenological research (MMPR), Mayoh and Onwuegbuzie (2014, 2015) may present the best pieces of recent literature that describe the relationship between research design and dominance. Although their primary purpose was not to reveal relationships between sequence and dominance, the analysis from Mayoh and Onwuegbuzie (2015) reveals patterns—some of which align with the literature above that states beginning with qualitative (phenomenological) methods tends to lead to qualitative-dominant research. However, they also challenge the conventional manner in explanatory designs where the initial quantitative methods allow such methods to dominate. Given these two findings, they suggest that the lack of quantitatively driven MMPR is because of the time consuming nature of qualitative inquiry. This is likely the case, yet by limiting their analysis to phenomenological studies (Moustakas, 1994), there is room for expanding examinations of sequence and dominance beyond studies that are inherently meant to be qualitative. Thus, using a case example—the social acceptance of wind energy literature—we move past the discussion surrounding how to combine methods (Creswell & Plano Clark, 2011; S. Hesse-Biber, 2010a) to reflect on how mixed methods are actually playing out in practice and to what extent method sequence may affect method dominance.

Research Context and Method

In this article, questions of method sequence and dominance are posed through an empirical analysis of the mixed methods, social acceptance of wind energy literature. We choose this literature because it is one in which we are practicing authors and is an appropriate size that can allow for such an in-depth investigation.

Due to the rise of public opposition to large wind turbines in rural communities, social scientific research into wind energy development has grown immensely over the past decade. In some cases, local stakeholder opposition has slowed the development of these renewable energy projects (see McRobert, Tennent-Riddell, & Walker, 2016), causing problems for developed countries who wish to reach their renewable energy and/or climate change targets (Batel & Devine-Wright, 2015). As in many domains of social scientific inquiry, early studies mainly used quantitative-based methods and found some evidence that individual-level traits—including selfish, not-in-my-backyard attitudes—were driving wind energy opposition (Krohn & Damborg, 1999). Years later, in recognition of the need for growth in this literature, Devine-Wright (2005) suggested that such explanations were simplistic and future work should consider a broader set of ideas and methodological approaches (see also Aitken, 2010). Ellis, Barry, and Robinson (2007) contextualized this point by stating that the reliance on quantitative methods alone “has contributed to the impasse in understanding of public perception of wind farms” (p. 540). Indeed, mixing methods may help us to incorporate multiple truths and “produce [both] the generalizable and the particular” (Warshawsky, 2014, p. 161). In response to such calls, we have seen a surge in qualitative and mixed methods work in what is now deemed the social acceptance of wind energy literature. This relatively sudden shift thus allows for a critical examination of a sizable and relatively recent collection of mixed methods research in one domain.

To gather a sample of mixed methods research in the area, we conducted two database searches—one in Google Scholar and the other in Web of Science using the Boolean search terms: “wind energy” OR “wind turbines” AND “mixed method” OR “mixed methodology” OR “qualitative quantitative” OR “q method.” In Google Scholar, this produced 1,640 journal articles and books published between 2005 and 2017. The sample dwindled to 18 after selection criteria were applied. An article was included in the final sample if it (i) was published in a peer-reviewed academic journal, (ii) was relevant to wind energy, (iii) was within the social sciences, and (iv) employed both qualitative and quantitative methods. Google Scholar has been criticized for gaps in coverage (Giustini & Boulous, 2013; Jacsó, 2005), so the journal database Web of Science was also used. With the same vetting as above, this search produced 16 new articles. Using this data set (N = 34), the following three questions frame this research:

Research Question 1: What is the sequence of qualitative and quantitative methods in each publication?

Research Question 2: Which method, if any, dominates the paper as a whole?

Research Question 3: How does method dominance potentially relate to method sequence?

For Research Question 1, the characterization of method sequence was based upon Holstein’s (2014) classification. We independently read through each paper to determine the research method order. In five instances, the method sequence was unclear so we contacted the author(s) and were able to confirm order in all of these cases.

To address Research Question 2 regarding dominance, we developed three analytic strategies. Together, we call these steps the Dominance in Mixed Methods Assessment (DIMMA) model (see Figure 1). We do this in the context that there is no generally accepted measure of dominance (Creamer, 2018; Creswell and Plano Clark, 2018).

Figure 1.

The three-step Dominance in Mixed Methods Assessment model used to measure method dominance in a mixed methods data set.

The first step was a three-stage interpretive reading of how the authors represented quantitative and qualitative data throughout each paper. This involved the first author reading through each paper in its entirety to qualitatively assess which method was prioritized. This subjective assessment looked at the following: (i) How the author(s) wrote about each method including the reasoning behind the use of each method (i.e., what Creswell and Plano Clark, 2011, call the “primary aim”), (ii) the amount of detail given about each method (through data collection, analysis), and (iii) the apparent quality and rigor of each strand (Baxter & Eyles, 1997). Following the examples of Bryman, Becker, and Sempik (2008) and Baxter and Eyles (1997), we use independent criteria for analyzing the apparent rigor of quantitative (e.g., validity, reliability, generalizability) and qualitative (e.g., credibility, transferability, dependability) separately. In six cases of doubt regarding the three assessments above, the second author also read each paper to increase intercoder agreement—a process by which multiple researchers come together to discuss coding discrepancies regarding the same text (Campbell, Quincy, Osserman, & Pedersen, 2013).

Second, a quantitative assessment of the amount of text devoted to each method in the Results section (using word counts) was performed. The technique is similar to the one Creswell and Plano Clark (2011) have outlined, yet it is difficult to find examples of its usage in practice. Although qualitative research is generally “richer” (Creswell, 2013)—requiring more space (i.e., higher word counts)—this more objective step is highly replicable to the extent that people agree (a) where a passage starts and ends and (b) that the content is either qualitative or quantitative. An example and full explanation of how this was performed can be found in Figure 2.

Figure 2.

An example of the results sections, word count process that makes up the second stage of the Dominance in Mixed Methods Assessment model (from Walker & Baxter, 2017b). Text highlighted in yellow indicates qualitative data, while text in blue indicates quantitative data. Text that could not be associated with either method would not be captured. After word counts were complete for both methods, a percentage calculation would be performed to determine the relative amount of qualitative findings in each paper’s results section. For example, using only the text above, we find 193 words devoted to qualitative results and 69 devoted to quantitative results. Thus, of the total word count 262, qualitative findings make up 73.7%.

Lastly, quantitative to qualitative sample size ratios were calculated for each paper to understand the relative size of the qualitative undertaking. While self-administered surveys can scale somewhat exponentially with minimal extra costs, each new interview tends to require the same effort and cost (Creswell and Plano Clark, 2011). Qualitative data sets most often included interviews (n = 29), but authors also chose focus groups, published text, and observation. Surveys (or questionnaires; n = 22) were the most common quantitative methods while some others chose sorts (i.e., in Q-method studies; n = 7), and other, sometimes vaguely described “economic modeling” or “quantitative analysis” data sets. Especially in comparison to one another, these sample ratios may suggest the amount of resources expended on each method in the overall research design (Onwuegbuzie & Collins, 2007). That is, though one would expect larger quantitative samples in most papers, relatively low ratios (surveys to interviews) may indicate higher levels of qualitative dominance. It should be noted that we did not attempt to quantify the size or length of each unit (n) from either the quantitative or qualitative data sets that made up each paper. Most often, these details were missing from each article, though it would be extremely helpful to differentiate, for example, between a set of 15 and 90-min interviews. We should be concerned about large sample qualitative studies if such sample sizes mean a sacrifice in depth of understanding.

After each step of the DIMMA process, we met as coauthors to review each paper and determine which method was dominant. Each step served important, if not equal roles through these discussions. In cases where no dominant method could be determined, we classified the paper as “neither” qualitative or quantitative. Together, these three approaches to deconstructing design and practice give a reasonable sense of method dominance—perhaps even beyond the conscious intent of the authors themselves.

Results

The findings are organized according to the three research questions and can be found in Table 1 (sequential designs) and Table 2 (nonsequential designs). The third question in particular is built upon the work of the previous two and culminates in the ultimate question of this research: What is the relationship between method sequence and method dominance?

Table 1.

Sequential Mixed Methods Articles^a (2005–2017) and Measures of Method Dominance.

Method	Author (Year)	Stated Purpose of Qualitative	% of Results With Qual.	Sample Ratio (Quan.:Qual.)^b	Dominant Method
Exploratory sequential	Walker and Baxter (2017a)	“used…to create a quantitative survey”	64.2	∼4:1 (240:54)	Neither
	Walker and Baxter (2017b)	“[to learn about] lived experiences [and]…to increase our understanding of procedural justice from multiple stakeholder perspectives”	60.3	∼4:1 (240:54)	Qual.
	Beckham Hooff, Botetzagias, and Kizos (2017)	“[to collect] various opinion types”	15.2	14:23 (28:46)	Quan.
	Nichifor (2016)	“[to gather] public perceptions of wind parks from the perspective of the respondents”	0	1:1 (104:104)	Quan.
	Walker, Baxter, and Ouellette (2015)	“allowed concepts to be developed inductively”	54.6	6:1 (152:26)	Qual.
	Fast, Mabee, and Blair (2015)	“[allows for] contingent meaning and interpretative flexibility”	32.8	∼2:1 (27:14)	Neither
	Brownlee et al. (2015)	“to develop a measurement instrument”	0	28:1 (483:17)	Quan.
	Fast et al. (2015)	“[to understand] residents’ impressions of how turbines fit or do not fit into landscapes”	41.2	13:1 (483:35+)	Qual.
	D’Souza and Yiridoe (2014)	“provide insight and understanding…and [help to develop] a survey”	28.1	38:1 (226:6)	Quan.
	Walker, Baxter, Mason, Luginaah, and Ouellette (2014)	“important insight” and “to inform and design the survey”	36.5	6:1 (152:26)	Quan.
	Walker, Baxter, and Ouellette (2014)	“triangulation of concepts…[and an] alternative…to purely positivistic approaches”	33.1	6:1 (152:26)	Neither
	Devine-Wright and Howes (2010)	“[triangulation and]…to create items in questionnaire”	25.1	14:1 (457:33)	Quan.
	Zoellner, Schweizer-Ries, and Wemheuer (2008)	“to understand the wide range of social parameters….”	53.2	N/A (349:N/A)	Qual.
	Ellis, Barry, and Robinson (2007)	“[to allow for the] study of attitudes”	17.4	5:1 (54:11)	Quan.
Explanatory sequential	Janhunen, Hujala, and Pätäri (2014)	“allowed a deeper understanding of the underlying factors”	37.2	8:1 (112:14)	Neither
	Lombard and Ferreira (2014)	“Triangulation”	0	N/A (98:N/A)	Quan.
	Schaefer, Lloyd, and Stephenson (2012)	“[to] identify perceived barriers and attitudes toward a [Feed In Tarriff]”	67.6	13:1 (366:29)	Neither
	Frantál and Kunc (2011)	“to investigate…socio-cultural contexts…and also the actual residents’ point of view”	26.1	2:1 (156:73)	Quan.
	Warren and McFadyen (2010)	“to explore the perceptions…concerning the impacts of onshore windfarms”	46.7	∼21:1 (106:5)	Quan.
	Holburn, Lui, and Morand (2010)	None given	42.9	N/A (29:N/A)	Quan.
	Maruyama, Nishikido, and Iida (2007)	“[to uncover] the interests of the various actors involved in community wind projects”	0	N/A (745:N/A)	Quan.
	Varho and Tapio (2005)	“as a test for sensibility…[and to reveal] otherwise hidden uncertainties”	41	1:1 (14:14)	Neither

^aThe full citations of each paper can be found within the references.

^bN/A refers to any sample (number) that was not stated in the paper. For example, some papers gave vague descriptions of the number of interviews they completed. If either a qualitative or quantitative sample was not given, a sample ratio was unable to be calculated.

Table 2.

Nonsequential Mixed Methods Articles^a (2005–2017) and Measures of Method Dominance.

Method	Author (Year)	Stated Purpose of Qualitative	% of Results With Qual.	Sample Ratio (Quan.:Qual.)^b	Dominant Method
Fully integrated	Frantál, Bevk, Van Veelen, Hărmănescu, and Benediktsson (2017)	“[to] provide an option for how to better capture and understand the subjective perceptions and preferences of people”	43.6	1:1 (30:30)	Quan.
	Edsand (2017)	“to allow experts to elaborate on their answers”	87.3	1:1 (16:16)^c	Qual.
	Mey and Diesendorf (2018)	“[mixed methods used to] broaden the understanding…[enable] triangulation [and] improve the validation”	96.8	N/A (N/A:14+)	Qual.
	J. S. Greene and Geisken (2013)	“to present a more complete picture”	49.8	8:1 (108+:12)	Quan.
	Jepson, Brannstrom, and Persons (2012)	“to explore the content and meaning of common views”	85.2	1:1 (21:11)	Qual.
	Brannstrom, Jepson, and Persons (2011)	“Create a concourse of statements [and]…elicit rationale (postsort)”^a	31.3	1:1 (21:11)	Quan.
	Wolsink and Breukers (2010)	“allows for…comparison of human subjectivity”	60.8	1:1 (56:56)	Qual.
	Fisher and Brown (2009)	“enables the researcher…[to study perceptions]”	50.1	1:1 (20:23)	Qual.
	Haggett and Toke (2006)	“to consider how protest manifests…[and] explore some of the issues raised”	50.8	N/A (51:N/A)	Neither
Convergent	Mulvaney, Woodson, and Prokopy (2013a)	“focused on assessing the benefits and costs…historical data…general concerns…and community involvement”	56.9	N/A (N/A:11)	Neither
	Mulvaney, Woodson, and Prokopy (2013b)	“a deeper understanding of the historical timeline and community acceptance of the wind farm”	25.0	N/A (N/A:N/A)	Quan.
	Maillé and St-Charles (2012)	“deepen the understanding”	48.4	1:1 (93:93)	Qual.

^aThe full citations of each paper can be found within the references. (Note: While the full citation for Mey and Diesendorf makes it clear it was fully published in 2018, it was available online in late 2017 and was gathered through our 2005–2017 search. Thus, after some discussion, it was chosen for inclusion in this study.)

^cIn addition to the interviews, there were history event analysis completed for this research. The number of documents analyzed (or otherwise n value) was not reported.

Research Question 1—Method Sequence

Using our “four families” categorization of research designs, we find there are more sequential (n = 22) than nonsequential (n = 12) designs. In particular, the exploratory sequential approach (n = 14; see Table 1) was the most prevalent within our sample. The explanatory sequential approach the second most common design (n = 8). That said there are a substantial number of fully integrated (n = 9) and convergent (n = 3) research designs in the case-study group of papers (see Table 2). Thus, from our sample, there is a variety of research designs (via method sequence) that are being used in the literature which allowed for more detailed analysis that followed.

Research Question 2—Method Dominance

Through subjective and objective measures to determine the overall method dominance in each paper (see DIMMA, Figure 1), there appears to be an overall theme of the subjugation of qualitative design and findings in the social acceptance of wind energy literature, but perhaps not on the same level as we might expect from the mixed methods literature. Of the 34 papers analyzed, nearly half (n = 16) were dominated by quantitative methods, while 10 were dominated by qualitative methods. There was also varying degrees of method dominance throughout the sample. For example, some papers (Brownlee et al., 2015; Lombard & Ferreira, 2014; Maruyama, Nishikido, & Iida, 2007) were heavily quantitative dominant, as shown by among other things, having only quantitative data within their results sections. Conversely, all papers that emphasized qualitative findings did so at a relatively less dramatic imbalance—with only three papers found to have more than 70% qualitative data (i.e., results section a word count) compared to 10 for quantitative dominant studies in the group. We detail levels of dominance (by method sequence) more thoroughly below, in Research Question 3—Relationship Between Sequence and Dominance subsection.

Research Question 3—Relationship Between Sequence and Dominance

Here, we expand upon the findings of measured dominance (see DIMMA model, Figure 1) and discuss whether method dominance is linked to method sequence. Overall, the explanatory sequential and fully integrated designs in particular showed the highest tendency to prioritize of quantitative and qualitative methods, respectively.

Exploratory sequential

Researchers who used the exploratory sequential approach (i.e., qualitative first) showed a preference for quantitative methods in their writing (Table 1). That is, of the 14 exploratory studies, 7 studies were dominated by quantitative reporting, 4 emphasized qualitative reporting, and the remaining 3 were balanced. Across the subsample of 14, most authors explain that the qualitative methods were used to “set up” or help design survey methods. Interviews were said to create measurement instruments (Brownlee et al., 2015; Devine-Wright & Howes, 2010) or to better inform “the more rigorous (quantitative) investigation” (D’Souza & Yiridoe, 2014, p. 264). Another reason for using mixed methodologies was to help overcome the complexity of the issues at hand. Zoellner, Schweizer-Ries, and Wemheuer (2008) cite their inclusion of qualitative interviews in particular as being vital because of “…wide range of social parameters that determine renewable energy processes in communities” (p. 4137).

The results sections of the exploratory papers are particularly indicative of method dominance. As shown in Table 1, there is no consistent pattern, but if there is any bias, it is toward the quantitative findings. The amount of space devoted to each method varies widely but equates to an average of 33% qualitative or a fairly strong preference for quantitative text—although this value is highly influenced by the studies of Nichifor (2016) and Brownlee et al. (2015) who devote none of their results section to qualitative findings.

In looking at sample sizes used for each method, we see similarities across the exploratory sequential research design. In most cases, the quantitative sample is much larger than the qualitative sample. Of the papers for which data are available, the ratio of quantitative to qualitative ranges from 38 to 1 (D’Souza & Yiridoe, 2014) to approximately 1 to 2 (Beckham Hooff, Botetzagias, & Kizos, 2017). Together with all of the subjective and objective measures used, we find there is a fairly strong preference for quantitative (i.e., survey) findings among exploratory sequential papers.

Explanatory sequential

Explanatory sequential (i.e., quantitative first) articles’ stated purpose for using qualitative methods avoided any mention of using one to inform or design the other. Instead, there was an indication that the qualitative methods were included to expand and delve deeper into research questions. That is, qualitative methods were used to allow for richer understandings (Janhunen, Hujala, & Pätäri, 2014), triangulation (Lombard & Ferreira, 2014), or to explore “residents” [actual] points of views” (Frantál & Kunc, 2011, p. 507). In these cases, interviews were most often used to further investigate findings that arose within the initial survey. There was also the paper by Varho and Tapio (2005) which suggested that conducting interviews after quantitative methods uncovered “interesting factors…which might not emerge through more formal methods” (p. 1945).

The findings sections within this subset of literature reveals that the authors who used an explanatory approach devoted less space toward the qualitative findings (32.7%). There was only one article which contained a majority (67.6%) of qualitative findings in its results section (Schaefer, Lloyd, & Stephenson, 2012). This trend is somewhat surprising considering how the qualitative methods were described above. The quantitative to qualitative sample ratios within the set of explanatory articles were also similar to those found in the exploratory sequential papers—although there were two with comparable ratios of 1:1 (Varho & Tapio, 2005) and 2:1 (Frantál & Kunc, 2011). Despite this indication that would suggest otherwise, overall, it is clear that explanatory sequential papers found in this study tend to prioritize the quantitative methods. That is, using both subjective and objective measures, five papers were found to prioritize quantitative methods and three presented the two methods in a somewhat balanced way.

Fully integrated

In all but one case of papers that used the fully integrated design, authors’ stated purposes for including qualitative methods centered on theoretical development or expansion. For example, J. S. Greene and Geisken (2013) used interviews to “present a more complete picture” (p. 4) and Frantál, Bevk, Van Veelen, Hărmănescu, and Benediktsson (2017) chose mental-mapping and open-ended questions to “provide an option for how to better capture and understand the subjective perceptions and preferences of people” (p. 235). The only exception to this rule was from a Q-Method paper in which Brannstrom, Jepson, and Persons (2011) used interview data to “create concourse of statements” to be used in a quantitative, sorting exercise.

In looking through the results sections of all integrated papers found, we calculate that qualitative findings make up a majority of the text (61.7%). This turned out to be the highest value found among all research designs studied. There is also a fair degree of consistency. Five of the nine articles contained between 43% and 61% qualitative findings. Interestingly, most of the “outliers” has qualitative methods dominate the results sections (i.e., 85–97%; Edsand, 2017; Jepson, Brannstrom, & Persons, 2012; Mey and Diesendorf, 2018).

The sample ratios found within fully integrated papers also reveal more preference given to qualitative methods. These ratios are approximately the same in six studies, one of which actually has a slightly larger qualitative sample (Fisher & Brown, 2009). Another unique feature found in this subset is the use of three or more separate methods of data collection. In the paper by J. S. Greene and Geisken (2013), economic modeling began the data collection, followed by in-depth interviews and finally surveys were sent to randomly chosen residents. Five more papers (Brannstrom, Jepson, & Persons, 2011; Edsand, 2017; Frantál, Bevk, Van Veelen, Hărmănescu, & Benediktsson, 2017; Jepson et al., 2012; Mey and Diesendorf, 2018) also employed at least three stages of data collection.

In all, the papers that employed fully integrated designs showed some tendencies to prioritize the qualitative data and findings. Five of the nine papers in this subsample were found to emphasize qualitative research. The other four were from Brannstrom et al. (2011), Frantál et al. (2017), and J. S. Greene and Geiksen (2013)—who emphasized the quantitative—and Haggett and Toke (2006) who presented each method equally.

Convergent

There were only three papers that used a mixed method convergent design. In two of these papers, the purpose for using qualitative methods was to deepen understanding (Mulvaney, Woodson, & Prokopy, 2013b; Maillé & St-Charles, 2012). In the other, the use of interview and document analyses was more specific—to “[assess] benefits and costs…historical data…general concerns…and community involvement” (p. 325).

Studies that used convergent designs had wide variations in terms of how much space was devoted to qualitative findings. The average of 43.4% is indicative of the fact that qualitative results served a somewhat complimentary role; however, this value was influenced heavily by one paper in particular (Mulvaney et al., 2013b) whose quantitative findings encompassed 75% of the results section. Only a single paper (Maillé & St-Charles, 2012) within this subset contained full details regarding sample sizes. Given the small numbers, there is no clear pattern or tendency for authors to prioritize one method or the other within convergent mixed methods designs. Our analysis concludes one paper emphasized the qualitative, another quantitative, and the third balanced the two.

Discussion

In light of the fact that some have “moved on” and now focus on the idea of methodological intent, this article has attempted to reinvigorate important discussions around method dominance in mixed methods research. It has introduced the DIMMA model—an analytical methodology for determining dominance—and has explored connections to sequence. With the social acceptance of wind energy literature as a case example, there is a relative balance in terms of research designs. Although most (22 of the 34) of the papers found were sequential—14 exploratory, 8 explanatory—more than one third were nonsequential (fully integrated [n = 9] or convergent [n = 3]). This variety suggests that researchers in this space are considering the many ways in which to employ both qualitative and quantitative methods. In doing so, academics highlighted here seem to be moving away from Morse’s (1991) 2-fold classification system which characterizes all mixed methods research as being conducted either simultaneously or sequentially.

One of the main messages from our analysis is that mixing methods does not necessarily lead to a paper that presents a balanced mix of findings. Of the 34 papers in our sample, we detected a lack of a dominant method in only 8. Although mixed methods do not necessarily imply balance (see Morse, 2010), and there may be good reason for emphasizing one type of data, it is nevertheless useful to interrogate how dominant quantitative methods (in particular) are and if there is bias in terms of methodological sequencing. This may be the first step in overcoming the “methodological orthodoxy” (S. Hesse-Biber, 2010b) of privileging quantitative methods in the majority of mixed methods research.

Somewhat in line with recent concerns that qualitative methods are only playing complementary roles (Harrison & Reilly, 2011; S. Hesse-Biber, 2016; Morgan, 2013; Teddlie & Tashakkori, 2012), we detected qualitative dominance in only 29% of the 34-paper sample. Meanwhile, nearly half (47%) were quantitative dominant. However, given the history of the social sciences, we expected an even larger difference in quantitative- to qualitative-dominant papers (Creswell et al., 2004; Harrison & Reilly, 2011; Plano Clark et al., 2008; McManamny et al., 2015; O’Cathain et al., 2007). In this sense, we conclude that qualitative methods exceeded our (low) expectations, which is encouraging for both the mixed method and qualitative research communities. A reason for this may be that those inclined toward conducting mixed methods are coming from somewhat stronger understandings of the value of qualitative research. Indeed, the “qualitative community” of mixed methods researchers are aware of the “qual-light” (Teddlie & Tashakkori, 2012) use of qualitative research and perhaps are emphasizing those methods more consciously. The collection of qualitative journals who are accepting of mixed methods research may also be a factor (see Archibald, Radil, Zhang, & Hanson, 2015).

Another key contribution of this article can be found through the development of a model for studying method dominance in mixed methods research. To date, the literature is inconsistent in terms of not only what research designs lead to qualitative versus quantitative dominance but also what methods tend to dominate in mixed methods research more generally (Creswell et al., 2006; S. Hesse-Biber, 2016; Mason, 2006; Maxwell et al., 2015). The lack of clarity is no doubt due to the lack of systematic assessments for studying method dominance. Inspired by the writings of others including Creswell and Plano Clark (2007, 2011), we develop the three-stage DIMMA model. We suggest each subjective and objective component of the model has value, and when applied to other literatures, it may increase our understanding of important methodological questions.

Regarding the interaction of method sequence and dominance in our case literature, there are two trends: (i) explanatory and exploratory (to a lesser degree) studies are associated with dominance of quantitative methods and (ii) research that uses the fully integrated approach tends to emphasize the qualitative portion. Given that our sample is only 34, we might best consider these tentative findings and working hypotheses to be explored further in other literatures. In looking at the effect method sequence may have on revealed dominance in a specific domain, this article adds to a very limited number of studies that have looked at the relationship. Traditionally, researchers have treated the two factors as independent (Hall & Howard, 2008) and thus may have ignored the possible interactive effects of sequence and dominance. The question of “why” remains—is it because quantitative researchers are drawn to these approaches or because of something inherent in the design (e.g., excitement over the relatively newer, “generalizable” findings in a project). More fodder for future research may come from the need to study the prevalence and impact of the instrument development design (see Harrison & Reilly, 2011) in mixed methods and whether author intention and method dominance have any relationship.

Mayoh and Onwuegbuzie (2015) present the best and most recent investigation into the relationship between sequence and dominance, though it is done within a domain that is inherently qualitative in nature. Similarly, recent research from Archibald et al. (2015) found strong prioritization of qualitative research in recent published research, but the sample was taken only from qualitative journals. In part to address this, our review was open to all mixed methods research in the broadly defined realm of the social acceptance of wind energy literature. Of course, there may be methodological preferences in this area as well, though perhaps less than was seen in the two aforementioned studies.

Despite our focus on method sequence, there are no doubt many other factors shaping methodological dominance in mixed methods research. Indeed, Bryman (2007) makes clear that researchers may intend or be “forced” to prioritize one method because of the many “predispositions and preferences” of researchers and funding agencies (p. 20). Based on the present findings, we suggest that research design (i.e., method sequence) should also be considered as a potential influence, though the degree to which it is so may require more in-depth research with authors themselves.

Four papers identified that they used qualitative methods but present only quantitative findings. These papers might have been omitted from analysis, but we included them to highlight that identifying mixed methods research can itself be challenging (Hurmerinta-Peltomäki & Nummela, 2006; Maxwell et al., 2015). Phrases like “based on interviews” or “insights from interviews” are perhaps meant to signal companion work published elsewhere rather than in the current paper. Less optimistically, these authors may be using “tokenistic” or “qual-light” methods simply to help their work stand out as a form of mixed methods inquiry (Ivankova & Plano Clark, 2018; Teddlie & Tashakkori, 2012). Whatever the case may be, the absence of significant qualitative work within them underscores two issues: (i) the need to be more explicit regarding broader study design and context and (2) the possibility to explore such scenarios in future, similar analyses to the one here.

As one might expect, the fully integrated approach tended to allow a relative methodological balance. Although the authors fail to highlight how they achieved that balance in any direct way, it may be that the design itself facilitates balance or simply be that authors who gravitate to this design tend to be more balanced. Future research could explore whether there are other aspects of the design and execution of the work that play an important role (Creamer, 2018).

Especially within studies that showed more quantitative dominance, there was often an implication that qualitative work holds secondary status regarding rigor or robustness (S. Hesse-Biber, 2010b). In those papers, authors tended to stress how the interviews were completed in order to feed into the design of quantitative measurement instruments (e.g., questionnaires, items for Q-sorts; Brownlee et al., 2015; Devine-Wright & Howes, 2010; D’Souza & Yiridoe, 2014). When researchers used qualitative research for theoretical advancement in our case sample, the methods tended to be much more balanced rather than simply qualitative focused.

Although we cover an entire literature, our N is 34, so a strict interpretation of the value of our findings may be largely hypothesis generating. That said, given the alignments we find in the current mixed methods literature in social science more broadly, we suspect our interpretations are not entirely particular to recent social acceptance of wind energy, mixed methods literature. Further, we have provided a transparent foundation for future work that might explore ideas about sequence, dominance, and design in larger and more diverse data sets. We also recognize that many of the publications analyzed here are part of larger research projects, in which there may be multiple publications and varied balances of qualitative and quantitative approaches (Tashakkori & Teddlie, 2010). Thus, we do not claim to be assessing the state of mixed methods within social acceptance of wind energy research as much as we are looking at the “mixed publications” that result from such research—a subtle, but important nod to factors like publication pressures (Bryman, 2007), which could bear further empirical scrutiny.

Conclusion

This article has provided a set of procedures for determining if and perhaps how qualitative methods are subjugated in mixed methods design, with specific reference to the relationship between method sequence and dominance. This allows a moment for researchers who use sequential mixed methods in particular to ponder whether allowing quantitative data to dominate fits with overall research goals and perhaps, whether the design predisposes researchers toward publishing as such. While there is not specific problem with quantitative-dominant publishing, we should reflect on what is lost by leaving qualitative findings on the “cutting room floor.”

Through an examination of a set of literature in detail, this article has also reminded the reader of the true value in conducting mixed methods research. As the use of mixed methods approaches becomes more and more common, it is important for academics to use mixed methods only when the research problem or question calls for it. As Fielding (2012) writes, “Rather than mixing because there is something intrinsic or distinctive about quantitative data or qualitative data, we mix so as to integrate the two fundamental ways of thinking about social phenomena” (pp. 125–126). That is to say despite all of the criticisms and complexities of mixed methods research presented here, there is still the potential for increases in our understanding of social scientific problems when using both qualitative and quantitative methods together. Especially when employed with a greater consideration for both approaches, researchers may be able to more fully and appropriately investigate social phenomena.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) received no financial support for the research,authorship,and/or publication of this article.

References

Aitken

(2010). Why we still don’t understand the social aspects of wind power: A critique of the key assumptions in the literature. Energy Policy 38, 1834–1841.

Archibald

M. M.

Radil

A. I.

Zhang

Hanson

W. E.

(2015). Current mixed methods practices in qualitative research: A content analysis of leading journals. International Journal of Qualitative Methods, 14, 5–33.

Arnon

Reichel

(2009). Closed and open-ended question tools in a telephone survey about “The Good Teacher” an example of a mixed method study. Journal of Mixed Methods Research, 3, 172–196.

Batel

Devine-Wright

(2015). Towards a better understanding of people’s responses to renewable energy technologies: Insights from social representations theory. Public Understanding of Science, 24, 311–325.

Baxter

Eyles

(1997). Evaluating qualitative research in social geography: Establishing ‘rigour’ in interview analysis. Transactions of the Institute of British Geographers, 22, 505–525.

Becker

Geer

(1957). Participant observation and interviewing: A comparison. Human Organization, 16, 28–32.

Beckham Hooff

Botetzagias

Kizos

(2017). Seeing the wind (farm): Applying Q-methodology to understand the public’s reception of the visuals around a wind farm development. Environmental Communication, 11, 700–722.

Brannstrom

Jepson

Persons

(2011). Social perspectives on wind-power development in West Texas. Annals of the Association of American Geographers, 101, 839–851.

Brownlee

Hallo

Jodice

Moore

Powell

Wright

(2015). Place attachment and marine recreationists’ attitudes toward offshore wind energy development. Journal of Leisure Research, 47, 263–284.

10.

Bryman

(2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6, 97–113.

11.

Bryman

(2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1, 8–22.

12.

Bryman

Becker

Sempik

(2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11, 261–276.

13.

Cameron

. (2009). The use of mixed methods research in VET research. Proceedings of aligning participants policy and pedagogy: Traction and tension in VET research: 12th AVETRA Annual Conference, April 15–17, Sydney, Australia.

14.

Campbell

J. L.

Quincy

Osserman

Pedersen

O. K.

(2013). Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research, 42, 294–320.

15.

Creamer

E. G.

(2018). An introduction to fully integrated mixed methods research. Thousand Oaks, CA: Sage.

16.

Creswell

J. W.

(2013). Qualitative inquiry and research design: Choosing among five approaches. Thousand Oaks, CA: Sage.

17.

Creswell

J. W.

Fetters

Ivankova

(2004). Designing a mixed methods study in primary care. The Annals of Family Medicine, 2, 7–12.

18.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods Research. Thousand Oaks, CA: Sage.

19.

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods Research. Thousand Oaks, CA: Sage.

20.

Creswell

J. W.

Plano Clark

V. L.

Gutmann

M. L.

Hanson

W. E.

(2003). Advanced mixed methods research designs. In Tashakkori

Teddlie

(Eds.), Handbook of Mixed Methods in Social and Behavioral Research (pp. 209–240). Thousand Oaks, CA: Sage.

21.

Creswell

J. W.

Shope

Plano Clark

V. L.

Green

D. O.

(2006). How interpretive qualitative research extends mixed methods research. Research in the Schools, 13, 1–11.

22.

Denzin

N. K.

(2010). Moments, mixed methods, and paradigm dialogs. Qualitative Inquiry, 16, 419–427.

23.

Devine-Wright

(2005). Beyond NIMBYism: Towards an integrated framework for understanding public perceptions of wind energy. Wind energy, 8, 125–139.

24.

Devine-Wright

Howes

(2010). Disruption to place attachment and the protection of restorative environments: A wind energy case study. Journal of Environmental Psychology, 30, 271–280.

25.

Doyle

Brady

A. M.

Byrne

(2016). An overview of mixed methods research–revisited. Journal of Research in Nursing, 21, 623–635.

26.

D’Souza

Yiridoe

. (2014). Social acceptance of wind energy development and planning in rural communities of Australia: A consumer analysis. Energy Policy, 74, 262–270.

27.

Edsand

H. E.

(2017). Identifying barriers to wind energy diffusion in Colombia: A function analysis of the technological innovation system and the wider context. Technology in Society, 49, 1–15.

28.

Ellis

Barry

Robinson

(2007). Many ways to say ‘no’, different ways to say ‘yes’: Applying Q-methodology to understand public acceptance of wind farm proposals. Journal of Environmental Planning and Management, 50, 517–551.

29.

Fast

Mabee

Blair

(2015). The changing cultural and economic values of wind energy landscapes. The Canadian Geographer, 59, 181–193.

30.

Fielding

(2012). Triangulation and mixed methods designs data integration with new research technologies. Journal of Mixed Methods Research, 6, 124–136.

31.

Fisher

Brown

(2009). Wind energy on the Isle of Lewis: Implications for deliberative planning. Environment and Planning. A, 41, 2516.

32.

Flick

(2017). Mantras and myths: The disenchantment of mixed-methods research and revisiting triangulation as a perspective. Qualitative Inquiry, 23, 46–57.

33.

Frantál

Bevk

Van Veelen

Hărmănescu

Benediktsson

(2017). The importance of on-site evaluation for placing renewable energy in the landscape: A case study of the Búrfell wind farm (Iceland). Moravian Geographical Reports, 25, 234–247.

34.

Frantál

Kunc

(2011). Wind turbines in tourism landscapes: Czech experience. Annals of Tourism Research, 38, 499–519.

35.

Gelo

Braakmann

Benetka

(2008). Quantitative and qualitative research: Beyond the debate. Integrative Psychological and Behavioral Science, 42, 266–290.

36.

Gibson

C. B.

(2017). Elaboration, generalization, triangulation, and interpretation: On enhancing the value of mixed method research. Organizational Research Methods, 20, 193–223.

37.

Giddings

L. S.

(2006). Mixed-methods research: Positivism dressed in drag? Journal of Research in Nursing, 11, 195–203.

38.

Giustini

Boulos

(2013). Google Scholar is not enough to be used alone for systematic reviews. Online Journal of Public Health Informatics, 5, 214.

39.

Glaser

B. G.

Holton

. (2007). Remodeling grounded theory. Historical Social Research [Supplement], 47–68.

40.

Glassner

Moreno

(Eds.). (2013). The qualitative-quantitative distinction in the social sciences (Vol. 112). Berlin, Germany: Springer Science & Business Media.

41.

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–274.

42.

Greene

J. S.

Geisken

(2013). Socioeconomic impacts of wind farm development: A case study of Weatherford, Oklahoma. Energy, Sustainability and Society, 3, 1–9.

43.

Guest

(2013). Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research, 7, 141–151.

44.

Haggett

Toke

(2006). Crossing the great divide—Using multi-method analysis to understand opposition to windfarms. Public Administration, 84, 103–120.

45.

Hall

Howard

(2008). A synergistic approach conducting mixed methods research with typological and systemic design considerations. Journal of Mixed Methods Research, 2, 248–269.

46.

Hanson

W. E.

Creswell

J. W.

Clark

V. L. P.

Petska

K. S.

Creswell

J. D.

(2005). Mixed methods research designs in counseling psychology. Journal of Counseling Psychology, 52, 224.

47.

Harrison

R. L.

Reilly

T. M.

(2011). Mixed methods designs in marketing research. Qualitative Market Research: An International Journal, 14, 7–26.

48.

Hesse-Biber

(2010a). Emerging methodologies and methods practices in the field of mixed methods research. Qualitative Inquiry, 16, 415.

49.

Hesse-Biber

(2010b). Qualitative approaches to mixed methods practice. Qualitative Inquiry, 16, 455–468.

50.

Hesse-Biber

(2016). Doing interdisciplinary mixed methods health care research: Working the boundaries, tensions, and synergistic potential of team-based research. Qualitative Health Research, 26, 649–658.

51.

Hesse-Biber

S. N.

Rodriguez

Frost

N. A.

(2015). A qualitatively driven approach to multimethod and mixed methods research. In The Oxford handbook of multimethod and mixed methods research inquiry. Oxford, England: Oxford University Press.

52.

Heyvaert

Hannes

Maes

Onghena

(2013). Critical appraisal of mixed methods studies. Journal of Mixed Methods Research, 7, 302–327.

53.

Holburn

Lui

Morand

(2010). Policy risk and private investment in Ontario’s wind power sector. Canadian Public Policy, 36, 465–486.

54.

Holstein

(2014). Mixed methods social networks research: An introduction. Mixed Methods Social Networks Research: Design and Applications, 1, 3–34.

55.

Hong

Q. N.

Pluye

(2018). A conceptual framework for critical appraisal in systematic mixed studies reviews. Journal of Mixed Methods Research, 1–15. Retrieved from https://journals.sagepub.com/doi/abs/10.1177/1558689818770058

56.

Hurmerinta-Peltomäki

Nummela

(2006). Mixed methods in international business research: A value-added perspective. Management International Review, 46, 439–459.

57.

Ivankova

N. V.

Creswell

J. W.

Stick

S. L.

(2006). Using mixed-methods sequential explanatory design: From theory to practice. Field Methods, 18, 3–20.

58.

Ivankova

N. V.

Plano Clark

V. L.

(2018). Teaching mixed methods research: Using a socio-ecological framework as a pedagogical approach for addressing the complexity of the field. International Journal of Social Research Methodology, 21, 409–424.

59.

Jacsó

(2005). Google Scholar: The pros and the cons. Online Information Review, 29, 208–214.

60.

Janhunen

Hujala

Pätäri

(2014). Owners of second homes, locals and their attitudes towards future rural wind farm. Energy Policy, 73, 450–460.

61.

Jeanty

G. C.

Hibel

(2011). Mixed methods research of adult family care home residents and informal caregivers. Qualitative Report, 16, 635–656.

62.

Jepson

Brannstrom

Persons

. (2012). “We don’t take the pledge”: Environmentality and environmental skepticism at the epicenter of US wind energy development. Geoforum, 43, 851–863.

63.

Kettles

A. M.

Creswell

J. W.

Zhang

(2011). Mixed methods research in mental health nursing. Journal of Psychiatric and Mental Health Nursing, 18, 535–542.

64.

Krohn

Damborg

(1999). On public attitudes towards wind power. Renewable Energy, 16, 954–960.

65.

Leech

Onwuegbuzie

(2009). A typology of mixed methods research designs. Quality & Quantity, 43, 265–275.

66.

Lombard

Ferreira

(2014). Residents’ attitudes to proposed wind farms in the West Coast region of South Africa: A social perspective from the South. Energy Policy, 66, 390–399.

67.

Lucero

Wallerstein

Duran

Alegria

Greene-Moton

Israel

… Schulz

. (2018). Development of a mixed methods investigation of process and outcomes of community-based participatory research. Journal of Mixed Methods Research, 12, 55–74.

68.

Maillé

M. È.

Saint-Charles

(2012). Social cohesion in a community divided by a wind farm project. Human Ecology Review, 19, 83–98.

69.

Maruyama

Nishikido

Iida

(2007). The rise of community wind power in Japan: Enhanced acceptance through social innovation. Energy Policy, 35, 2761–2769.

70.

Mason

(2006). Mixing methods in a qualitatively driven way. Qualitative Research, 6, 9–25.

71.

Maxwell

J. A.

Chmiel

Rogers

S. E

. (2015). Designing integration in multimethod and mixed methods research. In Hesse-Biber

S. N.

Johnson

R. B.

(Eds.), The Oxford handbook of multimethod and mixed methods research inquiry. Oxford, England: Oxford University Press.

72.

Mayoh

Onwuegbuzie

A. J.

(2014). Surveying the landscape of mixed methods phenomenological research. International Journal of Multiple Research Approaches, 8, 2–14.

73.

Mayoh

Onwuegbuzie

A. J.

(2015). Toward a conceptualization of mixed methods phenomenological research. Journal of Mixed Methods Research, 9, 91–107.

74.

McManamny

Sheen

Boyd

Jennings

P. A.

(2015). Mixed methods and its application in prehospital research a systematic review. Journal of Mixed Methods Research, 9, 214–231.

75.

McRobert

Tennent-Riddell

Walker

(2016). Ontario’s green economy and green energy act: Why a well-intentioned law is mired in controversy and opposed by rural communities. Renewable Energy Law and Policy Review, 7, 91–112.

76.

Mertens

D. M.

(2011). Publishing mixed methods research. Journal of Mixed Methods Research, 5, 3.

77.

Mertens

D. M.

(2014). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. Thousand Oaks, CA: Sage.

78.

Mey

Diesendorf

(2018). Who owns an energy transition? Strategic action fields and community wind energy in Denmark. Energy Research & Social Science, 35, 108–117.

79.

Molina-Azorín

J. F.

López-Gamero

M. D

. (2016). Mixed methods studies in environmental management research: Prevalence, purposes and designs. Business Strategy and the Environment, 25, 134–148.

80.

Morgan

D. L.

(1998). Practical strategies for combining qualitative and quantitative methods: Applications to health research. Qualitative Health Research, 8, 362–376.

81.

Morgan

D. L.

(2013). Integrating qualitative and quantitative methods: A pragmatic approach. Thousand Oaks, CA: Sage.

82.

Morse

M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40, 120–123.

83.

Morse

J. M.

(2010). Simultaneous and sequential qualitative mixed method designs. Qualitative Inquiry, 16, 483–491.

84.

Morse

J. M.

Niehaus

(2009). Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press.

85.

Moustakas

(1994). Phenomenological research methods. Thousand Oaks, CA: Sage.

86.

Mulvaney

K. K.

Woodson

Prokopy

L. S.

(2013a). A tale of three counties: Understanding wind development in the rural Midwestern United States. Energy Policy, 56, 322–330.

87.

Mulvaney

K. K.

Woodson

Prokopy

L. S.

(2013b). Different shades of green: A case study of support for wind farms in the rural midwest. Environmental Management, 51, 1012–1024.

88.

Nichifor

M. A.

(2016). Public reactions towards wind energy instalments. Case study: Romania and the Netherlands. Management & Marketing, 11, 532–543.

89.

Niglas

. (2004). The combined use of qualitative and quantitative methods in educational research (Doctoral dissertation). Tallinn Pedagogical University, Tallinn, Estonia.

90.

O’Cathain

Murphy

Nicholl

. (2007). Why, and how, mixed methods research is undertaken in health services research in England: A mixed methods study. BMC Health Services Research, 7, 1.

91.

Onwuegbuzie

A. J.

Collins

K. M.

(2007). A typology of mixed methods sampling designs in social science research. The Qualitative Report, 12, 281–316.

92.

Palinkas

Horwitz

Chamberlain

Hurlburt

Landsverk

(2011). Mixed-methods designs in mental health services research: A review. Psychiatric Services, 62, 255–263.

93.

Patry

J. L.

(2013). Beyond multiple methods: Critical multiplism on all levels. International Journal of Multiple Research Approaches, 7, 50–65.

94.

Plano Clark

V. L.

Ivankova

N. V.

(2015). Mixed methods research: A guide to the field (Vol. 3). Thousand Oaks, CA: Sage.

95.

Plano Clark

V. L.

Huddleston-Casas

C. A.

Churchill

S. L.

Green

D. O. N.

Garrett

A. L.

(2008). Mixed methods approaches in family science research. Journal of Family Issues, 29, 1543–1566.

96.

Schaefer

M. S.

Lloyd

Stephenson

J. R.

(2012). The suitability of a feed-in tariff for wind energy in New Zealand—A study based on stakeholders’ perspectives. Energy Policy, 43, 80–91.

97.

Schram

A. B.

(2014). A mixed methods content analysis of the research literature in science education. International Journal of Science Education, 36, 2619–2638.

98.

Seifert

Goodman

King

Magolda

(2010). Using mixed methods to study first-year college impact on liberal arts learning outcomes. Journal of Mixed Methods Research, 4, 248–267.

99.

Shannon-Baker

(2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10, 319–334.

100.

Sligo

J. L.

Nairn

K. M.

McGee

R. O.

(2017). Rethinking integration in mixed methods research using data from different eras: Lessons from a project about teenage vocational behaviour. International Journal of Social Research Methodology, 21, 63–75.

101.

Stentz

J. E.

Clark

V. L. P.

Matkin

G. S.

(2012). Applying mixed methods to leadership research: A review of current practices. The Leadership Quarterly, 23, 1173–1183.

102.

Tashakkori

Creswell

J. W.

(2007). Editorial: Exploring the nature of research questions in mixed methods research. Journal of Mixed Methods Research, 1, 207–211.

103.

Tashakkori

Teddlie

(Eds.). (2010). Sage handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage.

104.

Teddlie

Tashakkori

(2006). A general typology of research designs featuring mixed methods. Research in the Schools, 13, 12–28.

105.

Teddlie

Tashakkori

(2012). Common “core” characteristics of mixed methods research: A review of critical issues and call for greater convergence. American Behavioral Scientist, 56, 774–788.

106.

Trow

(1957). Comment on participant observation and interviewing: A comparison. Human Organization, 16, 33–35.

107.

Turner

S. F.

Cardinal

L. B.

Burton

R. M.

(2017). Research design for mixed methods: A triangulation-based framework and roadmap. Organizational Research Methods, 20, 243–267.

108.

van der Roest

J. W.

Spaaij

van Bottenburg

(2015). Mixed methods in emerging academic subdisciplines: The case of sport management. Journal of Mixed Methods Research, 9, 70–90.

109.

Varho

Tapio

(2005). Wind power in Finland up to the year 2025—’Soft’ scenarios based on expert views. Energy Policy, 33, 1930–1947.

110.

Walker

Baxter

. (2017a). “It’s easy to throw rocks at a corporation”: Wind energy development and distributive justice in Canada. Journal of Environmental Policy & Planning, 19, 754–768.

111.

Walker

Baxter

(2017b). Procedural justice in Canadian wind energy development: A comparison of community-based and technocratic siting processes. Energy Research & Social Science, 29, 160–169.

112.

Walker

Baxter

Mason

Luginaah

Ouellette

(2014). Wind energy development and perceived real estate values in Ontario, Canada. AIMS Energy, 2, 424–442.

113.

Walker

Baxter

Ouellette

(2014). Beyond rhetoric to understanding determinants of wind turbine support and conflict in two Ontario, Canada communities. Environment and Planning A, 46, 730–745.

114.

Walker

Baxter

Ouellette

(2015). Adding insult to injury: The development of psychosocial stress in Ontario Wind Turbine communities. Social Science & Medicine, 133, 358–365.

115.

Warren

C. R.

McFadyen

(2010). Does community ownership affect public attitudes to wind energy? A case study from south-west Scotland. Land Use Policy, 27, 204–213.

116.

Warshawsky

(2014). The potential for mixed methods: Results from the field in urban South Africa. The Professional Geographer, 66, 160–168.

117.

Wolsink

Breukers

(2010). Contrasting the core beliefs regarding the effective implementation of wind power. An international study of stakeholder perspectives. Journal of Environmental Planning and Management, 53, 535–558.

118.

Zoellner

Schweizer-Ries

Wemheuer

(2008). Public acceptance of renewable energies: Results from case studies in Germany. Energy Policy, 36, 4136–4141.

119.

Žydžiumaite

(2007). Methodological considerations: Sequential Linking of qualitative and quantitative research. Social Sciences (1392-0758), 55, 7–14.