Ensuring Rigor in Qualitative Data Analysis

Abstract

Deep and insightful interactions with the data are a prerequisite for qualitative data interpretation, in particular, in the generation of grounded theory. The researcher must also employ imaginative insight as they attempt to make sense of the data and generate understanding and theory. Design research is also dependent upon the researchers’ creative interpretation of the data. To support the research process, designers surround themselves with data, both as a source of empirical information and inspiration to trigger imaginative insights. Constant interaction with the data is integral to design research methodology. This article explores a design researchers approach to qualitative data analysis, in particular, the use of traditional tools such as colored pens, paper, and sticky notes with the CAQDAS software, NVivo for analysis, and the associated implications for rigor. A design researchers’ approach which is grounded in a practice which maximizes researcher data interaction in a variety of learning modalities ensures the analysis process is rigorous and productive. Reflection on the authors’ research analysis process, combined with consultation with the literature, would suggest digital analysis software packages such as NVivo do not fully scaffold the analysis process. They do, however, provide excellent data management and retrieval facilities that support analysis and write-up. This research finds that coding using traditional tools such as colored pens, paper, and sticky notes supporting data analysis combined with digital software packages such as NVivo supporting data management offer a valid and tested analysis method for grounded theory generation. Insights developed from exploring a design researchers approach may benefit researchers from other disciplines engaged in qualitative analysis.

What Is Already Known?

Deep and insightful interactions with the data are a prerequisite for qualitative data interpretation, in particular, in the generation of grounded theory.

What This Paper Adds?

Insights developed from evaluating a design researchers’ approach to qualitative data analysis which maximizes data interaction in a variety of learning modalities may support a more rigorous and productive analysis.

Introduction

This article reflects upon a design researchers approach to qualitative data analysis. The qualitative data analysis reflected upon formed part of a grounded theory study exploring the research process of designers, their understanding of and approach to research, with a view to developing a grounded theory to explain this process. The researcher who carried out the study is a designer and this is a reflection on their experience of qualitative data analysis while undertaking a grounded theory study. The objective of this article is to illustrate how tacit knowledge of multimodality forms of data interaction acquired in design practice and design research can support a rigorous and productive grounded theory analysis.

Method

This article will first outline and describe grounded theory analysis. Charmaz (2006) version of grounded theory is adopted here as it most closely aligns with a design approach to research enquiry incorporating pragmatism, symbolic interactionism, and an interpretivist view of research. The focus in this article will be on the nature and depth of interaction with the data required and the “creative crafting,” which is essential to grounded theory analysis. It is with these two elements of analysis that a design researchers tacit knowledge and experience of working with data may provide the most useful insights. The alignment between both research approaches and methods of analysis and interpretation will be described and mapped. The definition and support of rigor in qualitative analysis will frame this mapping and alignment.

This will be followed by an overview of the ongoing digital evolution where material tools, such as physical models, paper, and pens, are being replaced by computer-assisted tools and software in both sociological and design research and practice. This article will discuss the impact it has had on research enquiry and interpretation, in particular, the impact it has had on modes of interaction with the data afforded to the researcher. Deep and insightful interactions with the data are a prerequisite for grounded theory interpretation and theory generation.

Using the grounded theory case study as a primary source of evidence, these modes of interaction between researcher and data will be mapped for both traditional material supported and digitally supported methods of analysis. This will be discussed in relation to multimodality modes of learning and associated benefits for interpretation. The impact this has on imaginative data context exploration and theory development will also be mapped. Research participants in this study gave their fully informed consent to participate in writing before engaging with the project. Their names have been changed for the purpose of anonymity.

Grounded Theory

Glaser and Strauss initially conceived the method in 1967. Charmaz (2006), having learned from both Glaser and Strauss in California, wrote her own interpretation in 2006. Charmaz version of grounded theory is adopted here as it most closely aligns with a design approach to research enquiry incorporating pragmatism (foregrounding practice as a test bed for theory; Dalsgaard, 2014), symbolic interactionism, and an interpretivist view of research. In grounded theory, theory is developed from and grounded in data. It is important to stay close to the data, remaining open, and flexible to emerging insights. Fruitful analysis requires constant and meaningful interaction between the researcher and the data. Data, which may come from a variety of sources, will aid the building of theory grounded in the interpretations and actions of the research participants in their daily reality. Data sources may include, for example, interviews and focus groups, field notes and memos, research literature, and policy documents. It is important data provide rich detail and capture a range of perspectives to aid the development of theory.

Coding is the main analytic process in grounded theory. This means asking analytic questions of the data, categorizing segments of data with a short name (a code), and using these codes to sort and develop an understanding of what is happening in the social situation being studied (Charmaz, 2006, pp. 42–43). In grounded theory, data collection and analysis occur simultaneously. There is a “constant interplay between data collection and analysis” (Suddaby, 2006, p. 636) with data analysis directing subsequent data collection toward the emerging analytic issues. Constant comparison of data with data and data with codes is key to this process. In this process, the researchers’ role is central. Unlike positivist models of scientific research where the researcher maintains a degree of separation from the object of enquiry, in this interpretative model, “the researcher is considered to be an active element of the research process, and the act of research has a creative component” (Suddaby, 2006, p. 638). The creative aspect of grounded theory research is stressed by Corbin and Strauss (1990) where they claim that if the researcher simply follows the procedures without “imagination or insight into what the data are reflecting,…then the published findings fail.” They describe how this:

Creativity depends on the researcher’s analytic ability, theoretical sensitivity and sensitivity to the subtleties of the action/interaction [of the participants]. (Corbin & Strauss, 1990, p. 19)

This kind of creative insight requires what is sometimes described as constant “immersion” in the data or even “drowning” in the data (Suddaby, 2006).

Design Research

Research is an integral element of the design process, both as a source of information and inspiration (Sanders, 2005). Johnson (2003, p. 39) concurs with Sanders describing design research as:

inherently paradoxical, [it is] both imaginative and empirical…. Design researchers must go beyond what they can find: to see more than is visible, and to learn more than can be heard. Accordingly, design research is an act of imagination, just as much as design itself. Yet it must also be grounded in empirical evidence.

Design research shares many of the characteristics of the qualitative analysis process in grounded theory. These include constant interaction and immersion in the data, numerous iterations of data collection and analysis, with ongoing analysis guiding the next stage of data collection, and the necessity of creative interpretive insights grounded in the data. This process is supported by multimodality forms of data interaction that take place in a studio environment. Designers, for example, will use drawing as a way to explore ideas and ask questions of the data. A number of studies by Cross on the work practices of exceptional designers verify this process:

The architects also use their drawings as a means of thinking “aloud” or “talking to themselves”…The architect Richard MacCormac [is quoted] as saying “I use drawing as a process of criticism and discovery.” (Cross, 1996)

The studio environment is fundamental to the realization of this process. A study by Keller, Sleeswijk Visser, van der Lugt, and Stappers (2009) on designer interaction with data and visual material outlines how

the traditional tools and environment [studio] of designers are filled with rich visual material such as sketches, photos, models and collages. [He describes how they] collect visual material as part of their way of working,…as a source of inspiration,…that they personalise their physical environment and surround themselves with rich information sources,…that the body plays an important role in creativity,…large movements of the body are found to loosen the mind,…. And that designers who share a studio know about each others work through visible physical collections. The main observation was that designers really surround themselves with a rich collection of physical materials

which they interact with on a daily basis and that this interaction is an intrinsic element of their research and design process. Kenneth Grange, a designer who took part in Cross’s study, describes the target of the process as “reaching through to the concealed plums” (Cross, 2001). Daily immersion in the data is required because

it’s the little bits of inspiration, the little sorts of byways and unlikely analogies and things that eventually produce what you recognise as being the right thing to do. (Grange cited in Cross, 2001, p. 53)

It is this requirement to see beyond the obvious interpretations and solutions, to move past superficial readings to gain that creative insight into what the data is telling you that is so important in both design research and qualitative data analysis. In a design context

the working style is based on periods of intense activity [multimodality modes of interaction with the data], coupled with other periods of more relaxed, reflective contemplation. (Cross, 2001, p. 57)

The studio environment and display of reference material allows the designer to engage with research material on a variety of levels, micro- to macroviews and over a period of time with varying levels of interaction and engagement. Rigor is demonstrated by this depth of engagement that enables the designer “to reach through to the concealed plums” (Cross, 2001, p. 53).

Demonstrating Rigor in Research

It is important to clarify that the requirements for demonstrating rigor in design research and in grounded theory qualitative analysis vary from those required in quantitative studies. The requirements of reliability, replication, and validity generally associated with demonstrating rigor in quantitative studies are less applicable to qualitative studies. This is because they were initially developed for quantitative studies and their focus is mainly on measurement and the adequacy of the measures.

Trustworthiness is considered a more appropriate criterion for evaluating qualitative studies. In order to ensure the process is trustworthy, Guba and Lincoln (1989) propose the research should satisfy four criteria. They are credibility, transferability, dependability, and confirmability. Credibility ensures the study measures what is intended and is a true reflection of the social reality of the participants. There are many strategies to address credibility that include “prolonged engagement” and member checks. Transferability relates to the ability of the findings to be transferred to other contexts or settings. Because qualitative research is specific to a particular context, it is important a “thick description” of the particular research context is provided allowing the reader to assess whether it is transferable to their situation or not. Dependability ensures the process is described in sufficient detail to facilitate another researcher to repeat the work. This requires a detailed audit trail. Confirmability is comparable to objectivity in quantitative studies. Here, the goal is to minimize investigator bias by acknowledging researcher predispositions. Adherence to this framework by adopting strategies, such as those outlined, to address the individual criteria supports a rigorous research process (Holloway, 2008; Korstjens & Moser, 2017; Shenton, 2004). Details of the approaches used in this study to ensure the process is trustworthy are outlined in the “Project Background” section of this article.

In terms of design research, the traditional criteria for demonstrating rigor in research are also contested. Biggs and Buchler (2007) propose that rigor in design research as in literature research belongs to the process.

We say that the process was rigorous, and therefore validates the claim of the outcome. We would not say the outcome was rigorous. Therefore, if we consider practice-based methods, we might conclude that they must be rigorously undertaken.

They go on to say that

the validity of a method is the appropriateness of the process to provide a solution to the problem at hand.

This point is also echoed by Fallman and Stolterman (2010) where they relate rigor to fitness for purpose that has similar values to credibility in qualitative studies. Using an example of establishing rigor in one element of design research, “design exploration,” ¹ they suggest that

an important criterion is to what extent the design researcher is able to continue to “problem set” rather than “problem solve.” The process of design exploration should open up a critical and creative approach that challenges mainstream assumptions in design…. This means that rigour can only be measured in relation to how well the approach does open up a design space and less how well that is done.

The focus on process is important here. The process must facilitate interactions with the data that allow for and support creative insights. This is achieved in design research by working in a studio environment. Here, the design researcher surrounds himself with data in a variety of forms, visual representations, models, infographics, and so on, allowing for and supporting multimodality forms of data interaction over a period of time. Engagement may occur in this environment at a variety of levels, from deep dive to broad overview and during periods of intense work and reflection, both important for creative insights.

The Impact of Digital Tools on Researcher/Data Interaction

The introduction of digital tools to the design environment has impacted on researcher/data interactions offering many benefits but also limitations to the process. To evaluate this impact, it is necessary to look at modes of learning and associated interactions. Douglas and Nil Gulari (2015) claim we learn and understand the world by interacting with and experiencing things in the environment. The knowledge created during the interaction is dependent upon two forms of cognition, sequential, and relational. “Both forms of cognition are complementary and necessary.” They go on to explain citing Arnheim,

[Sequential is] situated in mathematics, for example where a mathematician follows a method of sequential progression as a means of solving a problem in which each step is accredited by the previous step and leads logically to the next in the chain. The second form of cognition is contextual and relational, situated in the way we experience colour, for example—we perceive the colour of an object in relation to its neighbours. (Douglas & Nil Gulari, 2015, p. 399)

He goes on to say both forms of cognition are complementary and necessary and are constitutive of cognition in everyday life. Generally, it is found that digital tools support a more sequential form of cognition whereas manual methods and tools support the relational. This is because digital tools can be restrictive when it comes to more relational forms of learning and interaction. People learn in varying combinations of visual, auditory, and kinesthetic modes (Birks & Mills, 2011, p. 103). However, digital (desktop computing) environments facilitate visual and kinesthetic modes in a limited way which may impact on the more relational modes of interaction with the data that are necessary for interpretation, contextualization, and reflection.

Research in design on the use of digital research and design tools has found limiting impacts on the more creative, interpretative, and reflective mode of cognition. Lawson, for example, on the use of computer-aided design (CAD) in design noted that designers using CAD when compared to designers drawing by hand

tended to persist with an idea for longer “vertically transforming” it. The inference here is that the less ambiguous [CAD] system allowed the designers less opportunity to “see” different interpretations of their drawings. As a result fewer ideas were explored in the process in roughly the same period of time. (Lawson, 2004, p. 71)

Lawson suggests that this might be because the “vectoring CAD systems use symbolic representations that do not map well onto the internal mental symbolic representations used by designers” (Lawson, 2004, p. 71). These points are echoed by Keller et al. who comment that “computer workflows force the designer into verbal mode: searching on keyboards, naming files and placing them in directories.” They go on to highlight the importance of “visual thinking,” “serendipitous encounters,” and “breaking the rhythm and involving the body” for creative exploration of ideas (Keller et al., 2009). James (2012) extending this critique of digital tools to the social sciences insists that “social science research is first and foremost a craft that involves the sociological imagination.” He cautions against using software in qualitative analysis stating that

Dealing with the blocks of often de-contextualized and disembodied data segments that computers can churn out may, if we are not mindful, lead us to forget the huge complexities of our subjects’ lives which, as analysts, we set out to understand. (James, 2012, p. 568)

Contextualization is an integral component of qualitative analysis. It is important to visualize the data from a range of perspectives. This is one of the main limitations of using computer software, as the user is often bound to the computer system context which is provided by the software. To explain, Birchfield et al. in a study of “learning contexts” links the “learning context” to our embodied interaction with the subject. They go on to say that

traditional HCI frameworks such as desktop computing (i.e., mouse/keyboard/screen) environments, which facilitate embodied interaction in a limited sense or not at all, risk binding the user to the system context, restricting many of his/her capacities for creative expression and free thought which have proven so essential in effective learning contexts. (Birchfield et al., 2008, p. 2)

This would suggest that it is important to vary the mode of interaction with the data, to shift from one perspective to another, in order to support creative insights and generate as complete a picture as possible of the phenomena we are trying to understand.

While acknowledging the importance of rich and meaningful interaction with the data, it is important not to forget the need to manage and organize a vast amount of data. There is a need to document the research methodology and data analysis procedure, to provide a transparent audit trail, and to communicate the rigor of the process adopted. The organization and management of data and its analysis are a prerequisite for the write-up and dissemination of the research undertaken. Digital software proves to be an invaluable tool for this process.

Grounded Theory Case Study

A grounded theory research project undertaken by the author provides a case study to evaluate modes of interaction afforded by traditional materials (colored pens, paper, sticky notes, and large format display boards) and the CAQDAS software package NVivo (Version 11) to support data analysis. Data management facilities will also be evaluated. The title of the research project is Articulating a Design Research Framework based on a Grounded Theory Approach.

Project Background

The social practice of design and design research is continually evolving to meet the needs of society. Designers who once directed their problem-solving ability on material artifacts are increasingly being called to address more complex social and environmental issues as part of collaborative multidisciplinary teams, increasing the role, and relevance of research in their profession. Research has always been an integral part of the design process; yet as a profession which developed outside of the university, its methodologies are fundamentally different from the more traditional academic models incorporating elements of creativity, intuition, and tacit knowledge. Increased collaboration with the wider academic research community, combined with greater focus on public research assessment and accountability, creates a clear need for design to develop, define, and communicate the research methodologies. Development of design research methodologies takes place within the contested space and value systems of academic research and design-led enquiry.

This research project aimed to explore the research process of designers, their understanding of and approach to research and to develop a grounded theory to explain this process. Data were collected mainly by means of qualitative semistructured interviews with practicing design researchers. A constructive grounded theory approach (Charmaz, 2006) was used for the qualitative analysis of the data. The researcher, coming from a design background, had experience in iterative constructive design research approaches and the similarities with grounded theory made it a logical and natural process to follow. Also, it was important theory developed from the study would be grounded in a designers understanding and approach to research. Grounded theory is an approach that facilitates this kind of understanding and theoretical development.

The purpose of reflecting on this case study is to focus on the authors’ interaction with the data during the analysis stage and the affordances offered by traditional materials and CAQDAS tools. To ensure the research process was trustworthy, Guba and Lincoln’s (1989) criteria for ensuring rigor in qualitative research were addressed by employing the following strategies.

For the purpose of credibility and to affirm the research measured a design researchers understanding of and approach to research, Charmaz, well-established methods of constructivist grounded theory research were followed. The constructivist version not only attends to the process being researched but also its embedded social and cultural context, the how and why of the participants’ understanding and actions. This, in turn, supports “thick description” and informed transferability of the research.

To support a constructivist approach, data was collected from a range of sources:

Interviews with 11 practicing design researchers. A theoretical approach to sampling was followed. Early interviews focused on academic design researchers. Ongoing analysis found that their research approaches varied and appeared to be influenced by their educational background. The next round of interviews selected participant design researchers from a range of educational institutions, art school, university, and technological university revealing further alignment between professional values and research approaches. To further explore this relationship and the range of variation in research approaches, the final interviews were conducted with research active practitioners working inside and outside of education.

Documentary analysis of official research evaluation publications, in particular, the UK REF 2014 to support understanding of how design research is represented and defined in the wider discursive and social practice of research assessment.

Examination of existing theoretical accounts relating to biography of design research practice and the social and cultural processes constitutive of its development. An awareness of the impact of historical and social structures and discourses was considered key to developing a contextual understanding of design research processes.

A constructivist approach acknowledges the interpretative nature of the findings. In order to monitor the researchers own developing interpretations and constructions, reflective journaling was conducted throughout the process. The researchers’ positionality, as a practicing design researcher in an Institute of Technology, and the bias it may generate, was also noted. Frequent debriefing sessions with research supervisors and peer review at conference further supported recognition of and attention to researcher bias. Finally, a comprehensive audit trail of all strategic decision-making, data gathering, and analysis was maintained.

To ensure the process of data analysis was rigorous, the researcher primarily followed Charmaz version of data analysis. This was combined with the use of Corbin & Strauss (1990, p. 13) coding paradigm to structure the affinity mapping process and provide a frame for focused coding. This process helped bring the fractured data together into a coherent whole and supported understanding of the relationships between categories. During the analysis process, annotations and memos were created recording the researchers developing interpretations of the data. These were recorded in a number of A4 hard-backed notebooks and in the NVivo software. Throughout the analysis procedure, both approaches to analysis were trailed before finally setting on a combined approach. This further supported prolonged interaction with the data from a range of positions.

For the purpose of this article, the following reflection will only consider the analysis of the 11 interviews with practicing design researchers. The documentary analysis of the UK REF 2014 and the theoretical accounts of the biography of design research practice will not be considered here, but they did provide ongoing dialogue with, and contextual understanding for, the analysis. Visualizations of both were created and displayed in the project work space. The interviews took place between the months of February and December 2015. Data analysis occurred at three different stages during the project and was interspersed with and guided the interviewing schedule.

Analysis of Modes of Interaction and Cognition During Coding

The researcher undertaking this study learned to code using a mixture of theoretical and practical guide books, master classes, workshops, and experiential learning. The experience of undertaking data analysis was where the greatest learning took place. During this process, the researcher coded the data using a variety of approaches. Continued reflection on, evaluation and comparison of these approaches informed the adaptation of a dual approach to qualitative analysis which combines CAQDAS (NVivo) with traditional materials of coding (colored pens, paper, and display boards). This was found to generate greater insights during the analysis process. Further visual analysis and mapping of modes of interaction and cognition afforded by the different approaches highlighted that the approaches which afforded greater modes of interaction and cognition increased the opportunity for interpretative insight leading to a more rigorous analysis procedure.

Coding Methods

The researcher trialed coding with and without the use of CAQDAS software before finally settling on a combined approach. This resulted in a number of interviews being coded more than once, encouraged reflection and comparison of emerging codes, particularly, codes which differed because of the coding approach adopted and ultimately increased the modes of interaction with the data. There were three approaches to coding in total:

Coding using A4 sheets of paper, colored markers, sticky notes, and large format display boards. Results recorded in photographs and captured in Microsoft Word Matrix (see Table 1 and Figures 1 –3; coded three interviews).

Digital coding with NVivo only (see Table 2; recoded one interview and coded two more interviews).

Digital coding with NVivo combined with coding using traditional materials: colored pens, paper, sticky notes, and large format display boards (see Table 3 and Figures 4 and 5). This approach was used to code two sets of three interviews.

Table 1.

First Approach—Open and Focused Coding Using A4 Sheets of Paper, Colored Pens, Sticky Notes, and Large Format Display Boards.

Coding DescriptionPrinted out the interview transcript on A4 sheets of paper leaving plenty of space between the lines of text and a wide margin for coding. Line-by-line coding was conducted manually with pens, markers, and sticky notes. The researcher highlighted in the text lines/phrases relating to the unit of analysis (designers doing research) and ascribed fledging codes in the marginsCoding ExampleThis coding and memo example is taken from an interview with David, an art school lecturer, research supervisor, and industrial design practitioner. See Figure 1 for image of A4 coding sheets with interview excerpt and codes, sticky notes, and memo notebook. In this interview, excerpt David is describing his PhD research process and the issues associated with not having a design research model to work from. He goes on to question the possibility of creating a model for design research given the intuitive/creative nature of the process and describes later how in his supervision of design research students, he encourages them to break the rules
Interview Transcript Excerpt“I would have modelled my approach very much on a scientific approach to PhD research rather than a design approach and more because I didn’t have a model to work from…I’m still unsure about how…how possible it is to model the creative process because so much of it relies on intuition and that kind of intuitive spark of energy that you know leads to creation and all the methodology in the world won’t necessarily bring you to that point, you know, it may allow you to understand it in hindsight but you know I think a straight jacket of any kind, of any kind, in a creative process could be a hindrance more than a aid you know and part of the, the kind of glory of creativity is freedom. Freedom to break the rules, to be able to work outside, to break new ground in a creative way”Fledging Codes Questioning Methodology:Noting a lack of design research models, questioning the possibility of “modeling the creative process,” viewing research methodology as being powerless to bring about creativity, and equating methodology with a “straight jacket,” and a “hindrance”Valuing Creativity: Seeing creativity as being reliant on “that kind of intuitive spark of energy,” Equating creativity with “glory” and “freedom,” equating creativity with “breaking the rules, working outside,” equating creativity with “breaking new ground in a creative way”Questioning methodology in its ability to support creativity:
Memo Title—Fundamental ConflictsDavid is reflecting on design research process in this excerpt. He notes the lack of design research models while questioning the possibility of modeling the creative process required for “breaking new ground in a creative way.” He sees a fundamental conflict between research methodology and the “kind of intuitive spark of energy” required “to break new ground in a creative way.” His use of language is emphatic on this point, for example, “all the methodology in the world won’t necessarily bring you to that point.” His reservations continue with equating research methodology with a “straight jacket” and a “hindrance” to the creative process. In contrast, creativity is associated with “glory” and “freedom,” “freedom to break the rules, to be able to work outside, to break new ground in a creative way.” The use of language is very strong in this excerpt reflecting fundamental beliefs/values and conflicts relating to the requirement for creative freedom in design research process and potential methodological constraints.Conflicts appear to relate to (research) process and (methodological) structure. They are expressed in terms of freedom and constraint, glory and dullness, spark and deaden, and energy and powerlessness. There are clear value and process differences and concerns regarding methodological structureAre all design researchers values/processes similar? How do other design researchers relate to methodology and structure?Open-Coding Description ContinuedThis process of coding continued until the entire interview was coded. During the process, emerging codes were compared with previous codes and amended if necessary to capture process and understanding. Memos continued to be written in a hard-backed notebook to record relationships between codes, ideas, and insights. A further two interviews were coded in the same mannerFocused or Axial CodingAt this point, all the fledgling codes from the three interviews were transferred to sticky notes and placed on a number of A1 sheets of paper (see Figure 2). This facilitated seeing relationships between codes within interviews and between interviews. Codes that seemed to be saying the same thing were grouped together with a pithy code from that group reflective of the core content being selected as a group heading. Memos continued to be written recording analytical reflections and decisionsTo add structure to this process, Strauss and Corbin’s (1990, 1998) coding paradigm was used. Here, codes were grouped under the following headings: (1) conditions/context (why, where, how, and what happens), (2) actions/interactions, emotions, and (3) consequences (of actions/interactions/emotions; (Birks & Mills, 2011, p. 96). The process allowed for imaginative exploration and reflection. The result was four A1 sheets of paper with codes on sticky notes for each interview. See Figure 3, for example. All 12 sheets were laid out on the table and floor in the room so all could be viewed at the same time (Data Interaction Mode: Visual and Kinesthetic). The method of “constant comparison” was practiced as the researcher compared codes with codes and categories with categories within interviews and between interviews (Cognition Mode: Sequential and Relational; Data Interaction Mode: Visual and Kinesthetic). Memos were written to describe the relationship between codes and categories. Further, rearranging was done until the researcher was content that the categories and codes best reflected the participants’ understanding and experience of doing research. Sticky tape was then used to fix the sticky notes to the sheets in the order they were arranged in. This would provide a visual record of the first round of analysis. Photographs were taken to record the process. A matrix was also created in Microsoft Word recording the categories and codes created Cognition Mode: SequentialData Management: Large format paper and interaction difficult to capture, except in photographs; however, the matrix in Microsoft Word captured the results of the analysis, if not the process. Process allowed for imaginative exploration and reflection
Coding ReflectionThe ability to see all the codes at once, to move them freely from one group to another and back again on large sheets of paper on a table, allowed free interaction with the data. Like a children’s card memory game, the researcher becomes familiar with all the codes, their actual, and possible positions in relation to their properties relative to their physical position on the sheets of paper (Cognition Mode: Relational; Data Interaction Mode: Visual and Kinesthetic). The physical layout also allows the researcher to reflect on the process as a whole and zoom in on smaller groupings, while in a reflective mode. With the addition of further interviews in this large viewing format, it was possible to compare codes with codes, categories with categories within interviews and between interviews. Furthermore, the large format sheets can be taken out, reflected upon, and compared with future coding and analysis
Cognition Mode	Data Interaction Mode	Data Management
Sequential Cognition ✓	Visual Mode ✓	Paper and Photographic capture ✓
Relational Cognition ✓	Auditory Mode ✓	Digital textual capture ✓
	Kinesthetic Mode ✓

Figure 1.

First approach—open coding: Photograph shows A4 coding sheets with interview excerpt and handwritten codes, sticky notes, and memo notebook.

Figure 2.

First approach—open and axial coding: Photograph shows sticky notes placed on large format display boards—Stage 1, affinity mapping process.

Figure 3.

First approach—open and axial coding: Photograph shows sticky notes arranged on large format display boards—Stage 2, affinity mapping process continued.

Table 2.

Second Approach—Open and Focused Coding Using NVivo Only.

Coding DescriptionThe researcher recoded one previously coded interview and coded two further interviews in sequence using NVivo software on the personal computer. This involves reading the interview text on the screen, highlighting key sections of text, and ascribing codes to these text sections in a sequential manner. Analytic questions and reflections related to text segments were captured during this process by using NVivo annotations. This function proved useful as the annotation and the text segment remained digitally connected and easily retrievable. Code memos written in NVivo during this process were also digitally linked with the code and the associated dataOn completion of three interviews, the researcher progressed to focused coding. This involved reviewing all the codes developed and grouping those that were reflecting similar actions and processes. A heading was selected to represent each of these core categoriesCoding ReflectionThe researcher found NVivo to be useful for data storage for recording connections, annotations, and memos but found it restrictive for data analysis, imaginative exploration, and reflection. The researchers design background supports more visual and kinesthetic work practices and felt limited by the computer work process format. For example, the computer screen size determines and limits how much of the interview and the emerging codes the researcher can see at any one time. This renders the process of constant comparison difficult and fails to encourage reflection. As a result, the researcher moved relatively quickly through the data, completed “open coding” and moved on to developing “core categories,” working at a more abstract level prematurely and without having fully considered the complexities of the participants stories. After some reflection, it was decided to combine both the colored pen and sticky notes method of analysis with NVivo to optimize the researchers’ interaction with the data, while maintaining a digital audit trail. It is important to note that NVivo was the only CAQDAS software trailed. Other packages may support a different experience
Cognition Mode	Data Interaction Mode	Data Management
Sequential Cognition ✓	Limited Visual Mode, Contextualization limited by computer system ✓	Paper and Photographic capture ✗
Relational Cognition ✗	Auditory Mode ✓	Digital textual capture ✓
	Kinesthetic Mode ✗

Table 3.

Third Approach—Digital Coding With NVivo and Traditional Materials (Colored Pens, Paper, Sticky Notes, and Large Format Display Boards) Coding Combined.

Coding DescriptionThis was the most satisfactory and fruitful analysis procedure. First, a new NVivo project was created with a new title. This was to limit the influence of the previous analysis and code names on this third round of analysis. It was important for the research that the researcher looks at the data with fresh eyes and from a fresh perspective. NVivo was then used to create codes for three further interviews in a number of sequential coding sessions. To look at the interviews with fresh eyes, to ensure coding was grounded in the data and that the researcher did not move too quickly into developing core categories or higher level abstractions, the researcher concentrated on developing codes which, where possible, reflected both the words of the participants and individual and collective processes. This strategy combines the use of In Vivo codes (participants’ actual words) with gerunds (coding for action and process). In Vivo codes help capture participants’ implicit meanings and understandings while the use of gerunds keeps the analysis active while supporting understanding of the relationships between meaning and action/process. For example, the In Vivo code “designers do see things differently” was initially selected from the interview excerpt below. This code had strong conceptual “grab” and could be linked with other text segments in the same and following interviews using the NVivo software, some of which are included below. It was subsequently elevated to a focused code (this happened at a later stage of analysis) and changed slightly to the gerund “seeing it differently.” It also had strong links and a codependency with another focused code “doing it differently” as can be seen in the interview excerpts“I would argue potentially that as a designer and a researcher I think designers do see things differently, they see patterns differently and opportunities emerging”“I think designers, just the way they are and it’s the way they operate, so I think they see the world differently and they’ll make patterns in relationships that maybe others wouldn’t”“maybe an engineer is looking for an optimum solution whereas designers are looking for something that’s a bit different you know to express themselves so yeah designers have something distinctive to offer”“So I went through a process, 7 engineers, these artists designed these benches, all very different. I think because I was a designer I could see the potential through this”“Again it’s seeing these opportunities…as a designer I could see things coming out of this and I could see how we could create some panels and plinths and exhibit it as cool stuff. So yeah and for me personally something coming out the end of it rather than a report you know”“In that research methodology record, we are different, we wouldn’t record endless notes in a lab unless it was particularly breakthrough, unless it was worth writing down”In Vivo coding was facilitated by the software package NVivo as the exact text from the interview could be highlighted and made into a code. Annotations and memos were created in NVivo during the process to record the analysis process and the rational behind the decisions made. This also encouraged the researcher to stop and reflect (Cognition Mode: Sequential and Relational)All the In Vivo codes developed in these coding sessions were then printed out and cut into strips and glued onto sticky notes. These sticky notes were then arranged, compared with each other, compared with earlier interview codes and transcripts, and rearranged using, as in round one, Strauss and Corbin’s (1990, 1998) coding paradigm and a large format display board. This is primarily where the focused or axial coding took place. As mentioned previously, the higher level codes were expressed where possible as gerunds derived from the In Vivo codes. Memos continued to be written developing the analytic process and reflecting on decisions made. A number of conceptual and visual maps were also used to support the analytic process. They further extend data interaction modes and provide a useful approach to exploring relationships within the study. For example, “doing it differently” became a core category. These “differences” were manifest in the design researchers’ values, processes, and situations/problems. The relationship between these and other variables were explored visually with paper and colored pencils. See Figure 4, for example, of a typical visualization and Figure 5 for a photograph of the affinity mapping process(Cognition Mode: Sequential and Relational; Data Interaction Mode: Visual and Kinesthetic), Once the researcher was satisfied the codes developed reflected the participants views, a digital matrix was created in Microsoft Word to reflect the findings. The process facilitated and encouraged constant comparison, imaginative exploration, and reflectionCoding ReflectionThe advantages of the combined process were the codes were initiated and recorded in NVivo along with their associated annotations and memos. This encouraged the researcher to stay close to the actual interview transcript as it is quick and easy to retrieve and it also helped maintain a clear data trail, while the interpretation, reflection, constant comparison, and so on were then further supported by the more interactive colored pens, paper, sticky notes, visual mapping, and large format display boards approach
Cognition Mode	Interaction Mode	Data Management
Sequential Cognition ✓	Visual Mode ✓	Paper and Photographic capture ✓
Relational Cognition ✓	Auditory Mode ✓	Digital textual capture ✓
	Kinesthetic Mode ✓

Figure 4.

Third approach—focused coding: Photograph shows a section of a visual exploring the relationship between the core category “doing it differently” and the manifestations of these differences.

Figure 5.

Third approach—axial coding: Photograph shows the In Vivo codes attached to the sticky notes and the affinity mapping process.

A detailed description of each coding process is presented in Tables 1–3. The three coding approaches are described in the coding description. The reflection discusses their ability to support visualizing the data from a range of perspectives and contextual settings as well as opportunities for imaginative exploration and reflection. This is followed by a summation of the mode of cognition (sequential or relational), mode of data interaction (visual, auditory, and kinesthetic) along with data management capability. It is important to note that auditory interaction with the data is beneficial. This occurred during the interview and by listening to the interview recording a number of times afterward to listen for meaning, review memos and field notes, and prepare the transcriptions. In this case study, auditory interaction with the data occurred in a similar manner in all three rounds of coding and subsequently is not discussed any further.

Discussion

The experience of data analysis combined with consultation with other researchers pointed to coding with colored pens, paper, sticky notes, and large format display boards as being the most beneficial form of coding, allowing the researcher great freedom in terms of constant comparison, trialing arrangements, viewing perspectives, reflection, and ultimately, developing interpretative insights. The physical act of writing on sticky notes, arranging sticky notes, rearranging them, visual mapping, and so on slowed down the process and encouraged a slower and more meaningful interaction with the data. It is important to keep all these manifestations of the analysis around the researcher, so a large workspace where visualizations, sticky notes, and concept maps may remain in place over a number of days is essential to this type of interaction process. This allows the researcher to engage with the research material on a variety of levels, micro- to macroview and over a period of time. It also supports peer-to-peer discussion and reflection of the analysis process. This researcher found the use of NVivo did not offer the same affordances. The computer screen is small and doesn’t facilitate broad overviews of the data, so the data views become fragmented. The researcher must call up the data they wish to see and so must make decisions based on memory rather than visually scanning the data. There were less “serendipitous encounters” and creative exploration of ideas and interpretations. These are encounters that might occur as the researchers scans all the data looking for relationships, connections, and so on. The advantages of NVivo are its data management facilities and its ability to generate answers to complex quantitative questions relating to the data. All data may be stored digitally on NVivo and quickly recalled. It complements working parallel to other coding methods. The preceding analysis and reflection on the coding process would suggest that this is because basic home computer software interactions limit visual and kinesthetic modes of data interaction. This, in turn, restricts the mode of cognition to mostly sequential with limited relational cognition. As previously stated by Douglas and Nil Gulari (2015), “both forms of cognition are complementary and necessary” for understanding in everyday life and, in this case, interpretative data analysis.

Conclusions

This article explored a design researcher’s approach to qualitative data analysis and the associated implications for rigor. A design research background supports more visual and kinesthetic work practices than those offered by the computer work process format and subsequently found it restrictive for analysis. A limitation of the research was that only one CAQDAS software package was trailed. Other packages may offer additional functionality. This is a topic for future studies.

Rigor in qualitative analysis belongs to the process and its trustworthiness. It is essential for the researcher to “immerse” themselves in data, to explore all the possible nuances and relationships, to view data from a variety of perspectives, and to move from micro- to macroview, in order to support the analytic imagination necessary for understanding and theory generation. This form of analysis is augmented by multimodality forms of interaction with the data. It takes time with periods of intense work followed by quiet reflection.

Reflection on this research analysis process combined with consultation with the literature would suggest digital qualitative analysis software packages such as NVivo do not fully scaffold the analysis process. Data interaction is limited by software design and screen size to a mainly sequential and constrained visual format, therefore not fully supporting the analytic and interpretative research processes. It does, however, provide excellent data management, quantitative analysis, and retrieval facilities which support the analysis and write up. This research finds that coding with sticky notes, colored pens, paper, and large format display boards, combined with digital software packages, such as NVivo, provide a valid and tested analysis method for grounded theory generation.