Sage Journals: Discover world-class research

Abstract

Objective:

Increasingly, people work in socially networked environments. With growing adoption of enterprise social network technologies, supporting effective social community is becoming an important factor in organizational success.

Background:

Relatively few human factors methods have been applied to social connection in communities. Although team methods provide a contribution, they do not suit design for communities. Wenger’s community of practice concept, combined with cognitive work analysis, provided one way of designing for community.

Method:

We used a cognitive work analysis approach modified with principles for supporting communities of practice to generate a new website design. Over several months, the community using the site was studied to examine their degree of social connectedness and communication levels.

Results:

Social network analysis and communications analysis, conducted at three different intervals, showed increases in connections between people and between people and organizations, as well as increased communication following the launch of the new design.

Conclusion:

In this work, we suggest that human factors approaches can be effective in social environments, when applied considering social community principles.

Application:

This work has implications for the development of new human factors methods as well as the design of interfaces for sociotechnical systems that have community building requirements.

Keywords

community cognitive work analysis communities of practice social networking social network analysis website design

Introduction

Human factors as a field has a strong history of developing methods and designs that improve human performance. Typically, human factors professionals work to analyze and redesign systems to help people take actions more quickly, more correctly, and with fewer errors. Although these goals are very important in technical and safety-related systems, systems with social components are becoming increasingly important. In social systems, the relevant measures of performance are whether people can connect effectively to the right people, interact with clarity, and make appropriate decisions on when and how to interact. During the design of systems, the consideration of social performance is becoming of equal importance to the consideration of task performance.

Despite the importance of social performance, few human factors methods have involved taking a serious look at it. Team methods have had the strongest influence, with a focus on the information needs of teams and individuals to improve team performance on a task (e.g., Cooke, 2005; Endsley, Bolte, & Jones, 2003; Naikar, Drumm, Pearce, & Sanderson, 2000; Salas, Dickinson, Converse, & Tannenbaum, 1992). Macroergonomics also involves stressing the importance of the organization in determining worker productivity and health (e.g., Cuevas, Fiore, Salas, & Bowers, 2002; Hendrick & Kleiner, 2002). The computer-supported collaborative work community has generated many interesting examples of how technology can change how people interact and work together (e.g., Christiansen, 2005; Malone & Crowston, 1990). There is also useful knowledge on social performance from the field of social psychology (e.g., Belbin, 1981; Tuckman, 1965). However, we believe there is an important gap between these communities and that there is a need for human factors methods to be able to analyze and generate design ideas that can be shown to influence social performance with the same rigor and reliability that human factors has brought to task performance.

A natural starting point would be the existing human factors methods that have a strong history of use in developing new designs that can improve human performance. There are several key methods (with many variants), including cognitive task analysis (e.g., Crandall, Klein, & Hoffman, 2006), situation awareness–oriented design (Endsley, 2011), and cognitive work analysis (Vicente, 1999). All of these methods have been effective at improving social performance. In particular, most of these methods have related “team” methods, such as team cognitive task analysis (Klein, 2000), team situation awareness (Endsley et al., 2003), and team cognitive work analysis (Ashoori & Burns, 2013). These team approaches focus on the information needs for teams as a whole, individuals working with those teams, and the communication between those individuals. These approaches present effective solutions for helping more tightly connected team-based groups perform well on complex tasks. By tightly connected teams, we mean groups or teams that work toward a common goal, share common tasks, and are expected to interact frequently with each other.

Communities are social systems that are much less tightly connected than teams. Members of communities typically interact less frequently than members of teams, and although community members may share common high-level goals, they may be working on a variety of different tasks. Although communities may not be driven or defined by preestablished deliverables like teams are, an effective community can generate several benefits to an organization. Over time, effective communities build practices and expertise within their members and promote the flow of information and practice between their members. The recent emergence of social networking technologies, particularly growing in the workplace, has made developing effective communities an important goal for organizational success. It is important to have human factors methods that involve considering community factors and recommending designs that have a measurable influence on the social effectiveness of communities.

The communities of practice (CoPs) concept provides recognition of these community factors. Communities of practice, as defined by Wenger, McDermott, and Snyder (2002), are “groups of people who share a concern, a set of problems, or a passion about a topic and who deepen their knowledge and expertise in this area by interacting on an ongoing basis” (p. 4). Although the CoPs concept provides a practical distinction to team-based groups, the concept by itself does not address issues of designing community supporting systems. We suggest that this approach presents an opportunity for integrating CoPs principles with a more design-relevant human factors method. We propose that a design built from a CoPs–human factors method should result in clear improvements in the effectiveness of a community over time.

In this work, we used a human factors method, cognitive work analysis (CWA) (Vicente, 1999), modified with principles from the CoPs concept, as developed by Wenger (1998) and Wenger et al. (2002), to design a new website for a community. We studied the influence of this new design on the community by taking a longitudinal approach, during which the community was examined before the new design was introduced, shortly after, and then several months later.

Method

In the following sections, we discuss the community under study, the design approach that was followed, and how the new design was evaluated in a community-relevant way.

The Community

Officially launched in 2008, the University-Community Partnership for Social Action Research (UCP-SARnet) is an organization based at Arizona State University with a mission to “educate, engage and empower communities” by facilitating global partnerships among universities, local governments, and community organizations (UCP-SARnet, 2012). These partnerships are aimed at creating new opportunities for joint action toward the realization of the United Nations Millennium Development Goals (MDGs).

UCP-SARnet maintains partnerships with 42 organizations and 971 members, including university students, university faculty, community activists, and members of local government. Seven volunteer “regional coordinators,” located in Argentina, Australia, Ghana, India, Nigeria, Poland, and South Africa, act as ambassadors for UCP-SARnet, facilitating collaborations between local community organizations, universities, and governments in efforts toward achieving the MDGs. The partners, members, and regional coordinators, led by a leadership team of 43 volunteers, work together to organize events promoting the MDGs, develop new strategic partnerships, and contribute directly to MDG-related projects.

To support its distributed activities, UCP-SARnet does much of its work using an online social networking portal powered by IGLOO Software (2011). The IGLOO platform provides a suite of features, including member profiles, blogs, discussion forums, wikis, calendars, document sharing, status updates, and email notifications. A site manager allows administrators of UCP-SARnet to customize the navigation and display of these features by using a drag-and-drop site manager and page editor. This combination of features allows UCP-SARnet administrators to choose how to stream member activities in any of these features through virtually any site structure and page layout. For instance, the administrator can stream new blog and discussion forum posts by dragging and dropping those components and arranging them on the page using the page editor.

In July 2010, UCP-SARnet began an effort to revamp its IGLOO-based social networking portal with the broad goal of making it “more attractive and alive.”

Design Approach

On the basis of the results of meetings with core members of the UCP-SARnet leadership team, as well as consideration of the intentions of CWA with the principles for supporting CoPs, a work domain analysis (WDA) (Vicente, 1999) was developed. A WDA is often used to model large, complex systems by representing functional components at varying levels of abstraction—from the concrete physical components up to the abstract functional purpose of the whole system (Vicente, 1999). When used as a basis for design, a WDA provides designers with a means of understanding and designing for systems at multiple levels of abstraction, which is helpful in systems with high complexity (Vicente, 1999). WDA is described more fully in Euerby and Burns (2012). Figure 1 shows the work domain model of UCP-SARnet. It should be noted that two views of the domain are presented, one specific to the purpose of the community (Domain) and the other related to the purpose of being a community (Community). In the left-hand column, traditional WDA levels are shown. The far right-hand column shows levels that were proposed to be more consistent with the principles for supporting CoPs. The key aspect of a community of practice is that it does more than work toward its stated goals (the left-hand column); it is also a social system that supports processes of interpersonal alignment, engagement, and imagination through the shared practices of the members (shown in the right-hand column of the model). We expect that a community that succeeds at building the components of the Community column will be sustainable while working toward, and after achieving, the United Nations MDGs and will be able to pursue further goals.

Figure 1.

View of the CoPs-inspired WDA, from Euerby and Burns (2012). CoPs = communities of practice; MDGs = Millennium Development Goals; WDA = work domain analysis.

The overall result of taking the CoPs-inspired approach was consideration of community building as a purpose in and of itself, which occurs simultaneously with the functional purposes of the community in the larger context.

New Design

As a result of the analysis, several changes were made to the design of the website. These changes are summarized as follows and discussed in more detail in Euerby and Burns (2012). Figure 2 shows the UCP-SARnet website and its main components before the redesign. Figures 3 and 4 show the main pages of the UCP-SARnet website after the redesign. Table 1 follows with a summary of the key changes to the design of the UCP-SARnet website.

Figure 2.

UCP-SARnet homepage screenshot before design changes.

Figure 3.

UCP-SARnet front page after design changes. Note that the lower half has been cropped due to space constraints.

Figure 4.

UCP-SARnet members area after design changes. A middle portion and footer have been cropped due to space constraints.

Table 1:

Summary of Key Design Changes and Link to the Analysis

Components	Design Concept	Community Support	Level of the WDA
Homepage	External facing page focused on presenting activities and successes of UCP-SARnet’s efforts toward MDGs	Staying current in the external communications provides members with a way to view and reflect on UCP-SARnet’s role in meeting the MDGs. In doing so, members may begin dialogue to refine and come to alignment on the CoP’s role in the larger context of meeting the MDGs.	Desired state of the learning community
	Congregates information and artifacts relevant to meeting the MDGs
	Provides a space for communication to prospective members and partners, as well as other stakeholders, in the larger context of the MDGs
Photo banner rotator	Visually displays specific efforts toward MDGs	The visual images of how UCP-SARnet is supporting the MDG efforts aids in members identifying new ways they may provide value in support of the MDGs.	Practices
Photo banner rotator	Presents examples of member participation		Practices
UCP-SARnet in Action blog	Provides a communication stream aimed to potential members, partners, and other stakeholders in the larger context	Allowing members to monitor the outgoing communication helps create an orientation toward providing value in the external context. Practices	Practices
		This orientation affects the dialogue in the members area toward identifying opportunities to provide value in the meeting of MDGs and reveal gaps in the knowledge of how to provide that value.	Practices
		Focusing the design on transient kinds of communication features (blogs) rather than static communication features (wikis) was a trade-off of reification for participation. This trade-off was seen to best serve alignment, engagement, and imagination in UCP-SARnet.	Trade-offs and priorities in the negotiation of meaning
Members area	Internal facing page focused on content authoring, sharing, and exchanging dialogue	Focusing on content authoring, sharing, and dialogue helps to give members experience with using the knowledge within the domain of the MDGs’ pursuit, so over time, UCP-SARnet members may learn what knowledge should be shared and how to share it.	Practices
	Provides a central space for informal dialogue about the MDGs and UCP-SARnet as a CoP		Practices
	Congregates internal information and artifacts relevant to the internal operations of UCP-SARnet	Setting a more informal context for dialogue helps members build the strength in relationships to actively discuss differences with respect to the domain and practice.	Desired state of the learning community
Members Area blog	Provides a communication stream for internal dialogue about the MDGs and how the CoP can improve its efforts toward them	Providing a flexible and informal medium for internal dialogue supports all of the practices.	Practices
Recent Activity bar/Latest Comments bar	Provides a historical view of the internal communication streams and comments as well as the exchange of documents and other artifacts	Providing a historical view of the dialogue allows members to better track conversations in such a way that helped in learning what knowledge should be shared and how.	Practices
	Encourages revisiting the site by showing what’s new and allows members to catch up with any unfolding dialogue	Encouraging message reciprocation inclined the core team to build a variety of connections through shared content matter, personal exchanges, and so on.	People and relationships
	Encourages message reciprocation by making comments more visible to the whole CoP		People and relationships
Facilitator list	Gives recognition to facilitators of their role in supporting the internal activities of UCP-SARnet	By giving recognition to these shared roles and their similar relationship to the UCP-SARnet CoP, this component helps to build connections between facilitators.	People and relationships

Note. CoPs = communities of practice; MDGs = Millennium Development Goals; WDA = work domain analysis.

Evaluation Objectives

An evaluation of the design interventions was conducted to determine the effect of the redesign of the UCP-SARnet portal. The evaluation was directed at determining if the new UCP-SARnet website increased the connectivity level of the UCP-SARnet community. This evaluation was consistent with the client’s goals in developing the website and featured prominently throughout the WDA at all levels. Furthermore, in this analysis approach, we were testing whether we had contributed to the goals of the design, rather than testing the individual design features. The three evaluation measures used were the following:

The website should increase the strong connections between members.

The website should increase connection with external organizations.

The website should encourage an increase in communication.

Figure 5, which is a reduced version of the UCP-SARnet WDA in Figure 1, lists the WDA elements evaluated in the analysis.

Figure 5.

Elements of the Work Domain Analysis in Figure 1 that were evaluated in the analysis.

Evaluation

The influence of the new website design was evaluated longitudinally with three evaluation points. Figure 6 shows the evaluation timeline. In the timeline, a baseline measure was taken before the new site was implemented (at or around January 1, 2011). The new site was launched on January 16, 2011. Then the effect of the new site was measured at two points, one about 2 months after the new site went online (around March 12, 2011) and one about 5 months after the new site went online (around May 29, 2011). In between the evaluation points and 2 months before the first survey point, the number of content posts, unique blog posts, and unique bloggers and commenters was captured. The intent of this approach was to try to differentiate transient changes in user behavior that may have occurred due to a spike in interest resulting from the launching of the new site and the associated advertising, from longer term, more sustainable changes resulting from the redesign of the site. The 6-month period was reasonable for examining more sustainable changes given the pace of change in website design cycles. For example, industry recommendations suggest that, with the current pace of technology, websites should be evaluated every 3 months and significantly reworked every year (AllBusiness, 2012). Figure 6 shows a timeline of the evaluation and when responses were received, as well as the distribution of survey responses over the 6-month period. The semi-transparent circles represent individual survey submission dates of the base survey respondents. The triangle above each cluster of circles indicates the average respondent submission times for each survey. Analytics software capturing the number of posts ran continuously throughout the study period from December 1 to June 29.

Figure 6.

Timeline of evaluation.

Surveys

Three identical surveys were conducted with the UCP-SARnet leadership team at each evaluation point. The surveys were focused on examining two aspects: the building of new connections between the UCP-SARnet community members and the building of connections between UCP-SARnet members and other organizations. As such, in the surveys, one question asked about the relationship between UCP-SARnet members and another question asked about the relationship between UCP-SARnet members and other organizations. The survey included participant identification information that was later coded for anonymity. There were also questions on the overall user experience with the website, which will not be reported in this paper.

Of the 43 members on the UCP-SARnet leadership team, there were 26 respondents in the first survey (S1), 24 respondents in the second survey (S2), and 17 respondents in the third survey (S3). The leadership team, made up primarily of students, gained new members and lost members over the course of the surveys, so there were differences in who was surveyed in each period. Between the three surveys, 14 base respondents completed all three surveys. Because of these differing numbers, the following analyses were conducted on data from the base respondents who completed all three surveys.

Social Network Analysis

We used social network analysis (SNA) to assess community structure and connections (Pfautz & Pfautz, 2009). SNA is a way for researchers to investigate interdependencies between actors and not just the actors themselves. It is also a method to map relationships among people, organizations, or other entities (Scott, 1992). In an SNA, the entities under study are referred to as nodes, and the relationships or information flow between nodes are referred to as edges or links. The visual representation of SNA maps shows the importance of each node, the amount of connectedness, and the strength of the relationships among nodes. How the researcher defines the relationship he or she is interested in depends on the scope and concerns of each particular research effort. Ultimately, the relational information of SNA goes beyond generalized attributes of the population and provides information about how a network of actors functions in relation to one another as a part of the whole.

With respect to the CoPs-inspired WDA shown in Figure 1, an SNA helped to provide a description of a CoP in both the domain and community dimensions. In the community dimension, there was an interest in understanding the quality of internal relationships between members of UCP-SARnet. In the domain dimension, there was an interest in understanding the external relationships between members of UCP-SARnet and surrounding organizations. To investigate these dimensions using SNA, we created a network map for the relations between members of UCP-SARnet and a network map for the relations between members of UCP-SARnet and external organizations. Each map was drawn at each survey time.

Actor-actor social network analysis

The data for the actor-actor SNA were gathered using a survey question that asked each respondent how well he or she knew other members of the leadership team. Each respondent was presented with a list of the other members of the leadership team and could choose a response from a 3-point rating scale: don’t know this person, know somewhat, and know well. This question was intentionally vague in terms of how “knowing someone” was defined, and the respondents were encouraged to answer with their gut feeling. The question was framed to appeal to the respondents’ intuitive sense and provide some measure of the elements of “building connections” and “strengthening relationships,” which were identified from the CoPs-inspired WDA (in the Community column of Figure 1).

For each of the three surveys administered, the data were processed into a list of relationships (don’t know, know somewhat, or know well) between a source actor and a target actor (both represented by a three-digit actor identifier), which was imported into the SNA software. Many varieties of SNA software are available for use, and Cytoscape was chosen for its flexibility and quality of rendering network graphs, as well as its ability to calculate standard SNA statistics, such as the network density, number of multi-edge node pairs, and average number of neighbors. For this study, edge data, made up of a list of source nodes, interaction types, and target nodes, was uploaded to Cytoscape. Source nodes corresponded to the respondent actors. The interaction type corresponded to the know-well connection to a target actor. With these data, a graph consisting of actor nodes and interaction edges was generated. The graph was a directed graph in the sense that it was possible that one actor could state that he or she knew the target actor well, whereas the target might not state knowing the source actor well.

The hypothesis for actor-actor relationships was as follows:

Hypothesis 1: The redesign of the UCP-SARnet social networking portal will increase the number of personal connections between members of the UCP-SARnet leadership team over time. Measurements of connections at S2 and at S3 will show increased connections over S1.

Actor-organization social network analysis

The second component of the SNA consisted of an actor-organization analysis to see how members of the UCP-SARnet leadership team were connected to the organizational partners. External networks and connections had been identified as domain elements in the CoPs-inspired WDA (Figure 1). The data for the actor-organization SNA were gathered using a survey question that asked respondents what level of connection they had with each of the UCP-SARnet partners. The respondents chose from a 3-point rating scale: strong connection, weak connection, and unaware. Unlike the actor-actor survey question, this question was given more concrete definitions for each of these responses. A “strong” connection was defined as a member of UCP-SARnet reporting having met or communicated with a member of the organization. A “weak” connection was defined as a respondent having heard of the organization but not met or communicated with anyone from that organization. “Unaware” was defined as a respondent not having heard of the organization at all.

In the same way as the actor-actor analysis, the data were organized into a list of relationships (strong connection, weak connection, or unaware) between source actor and target actor (three-digit actor identifier), which was imported to Cytoscape. The resulting graphs were undirected graphs with the assumption that the level of connection stated by the respondents would be the level of connection that the organization also would have stated if surveyed.

With regard to the actor-organization SNA, the hypothesis was as follows:

Hypothesis 2: The redesign of the UCP-SARnet social networking portal will increase the number of connections between the UCP-SARnet leadership team and the UCP-SARnet organizational partners over time. Measurements of connections at S2 and at S3 will show increased connections over S1.

Website Communication

Communication was examined in terms of the amount of activity, not the content of the activity. The intention behind this measure was to see whether the new site was given encouragement in posting content from a distribution of users. The website communication analysis provided an examination of the blog and comment posts made on the UCP-SARnet web portal before the first survey period and before the next two survey periods. The communication analysis consisted of a count of the blog posts, comment posts, and unique bloggers and commenters over each week. These metrics were chosen to provide a view of how the UCP-SARnet members were using the redesigned UCP-SARnet website, which featured a community blog stream.

For the count of posts to the website, the main hypothesis was as follows:

Hypothesis 3: The redesign of the UCP-SARnet social networking portal should increase the number of blog posts, comments, and unique weekly posters to UCP-SARnet over time. In particular, activity should be greater after the launch of the website, and in the final period, activity should still be greater than in the first measured period before the new website was launched.

Results

The results are presented in three sections. Data Analysis provides an outline of the statistical calculations and tests used, followed by a description of the results of the actor-actor SNA, the actor-organization SNA, and communications analysis. As mentioned previously, only respondents who completed all three surveys were included in the SNA.

Data Analysis

To test Hypothesis 1, actor-actor social network graphs of directed know-well connections were drawn and analyzed to view the new connections and how those connections were distributed throughout the base respondents. Using the count of the know-well connections for each respondent, we conducted a repeated measures analysis of variance (ANOVA) over each survey to determine if there was a significant increase in the number of know-well connections over time. Given a significant result of the repeated measures ANOVA, post hoc tests were conducted to compare the mean number of connections between S1 and S2, S2 and S3, and S1 and S3 to determine between which surveys the significant changes may have occurred.

As a complement to the repeated measures ANOVA, the network density of the know-well networks was calculated for each survey time. The network density is a ratio of the number of edges (recorded as know-well connections between actors) in the network divided by the number of possible pairs in the network. The density measure can help to determine the extent of the changes to the network—a higher change in density would correspond to a stronger effect of the website redesign. Network density would be calculated using the following equation:

$Density = \frac{Number of edges}{Number of possible pairs pairs},$

where the number of pairs in a directed graph is calculated using the equation

$Number of pairs = (number of nodes) * (number of nodes - 1)$

To test Hypothesis 2, the actor-organization social network graphs of undirected “strong” connections were drawn and analyzed to view the new connections and how they were distributed throughout the base respondents. Using the count of strong connections for each respondent, a repeated measures ANOVA was conducted over each survey to determine if there was a significant increase in the number of strong connections over time. Given a significant result of the repeated measures ANOVA, post hoc tests were conducted to compare the mean number of connections between S1 and S2, S2 and S3, and S1 and S3 to determine between which surveys the significant changes may have occurred.

As a complement to the repeated measures ANOVA, the network density of the strong networks was calculated for each survey time. As in the actor-actor network, the network density measures would help to determine the extent of the changes to the network—a higher change in density would correspond to a stronger effect of the website redesign.

Because the actor-organization network was undirected, the number of possible pairs was half as many:

$Number of pairs = \frac{[(Number of nodes) * (Number of nodes - 1)]}{2} .$

Furthermore, in contrast to the actor-actor network, the number of nodes increased or decreased as respondents drew new connections or dropped connections with more organizational partners over each survey. Therefore, in the calculations of density, it was not a simple increase in edges that would increase density, but the calculation also had to factor in the change in the number of nodes in the networks over time.

Only the know-well and strong connections were used in the actor-actor analyses and the actor-organization analyses because they were seen as the best indicators of successful improvements to the website. Analyzing the middle and lower levels of the survey could provide useful information as well but could reflect individuals forming or shedding relationships. In contrast, the know-well relationships should be more stable.

To test Hypothesis 3, blog posts, comments, and unique commenters and bloggers were counted during three periods of 10 weeks corresponding to the survey dates: Period 1 (P1), 10 weeks before the release of the website on January 16, 2011, corresponding to S1; Period 2 (P2), 10 weeks leading up to the end of S2; and Period 3 (P3), 10 weeks leading up to the end of S3. A multivariate analysis of variance (MANOVA) was conducted with the weekly blog posts, comments, and unique weekly posters to see if there was an effect of the period on the frequency of communications on the site. This broad MANOVA was followed up with multiple univariate ANOVAs for each of the three communication measures to determine which communication measures were significant over the three periods. For the measures with significant univariate ANOVAs, post hoc Tukey tests were used to determine between which period the significant effects may have occurred.

Actor-Actor Social Network Analysis

Figure 7 shows the actor-actor SNA for the 14 base respondents of the UCP-SARnet leadership team at S1, S2, and S3. These graphs show the know-well connections between the respondents at the time of each survey and are accompanied with some basic statistics for comparison. Figure 8 presents graphs of the new know-well connections created between S1 and S2, S2 and S3, and S1 and S3.

Figure 7.

Actor-actor social network analysis results. The circle nodes contain random numbers that correspond to the 14 survey respondents.

Figure 8.

Difference graphs for the actor-actor social network analysis. The circle nodes contain random numbers that correspond to the 14 survey respondents.

In Figure 7, the visual structure of each graph was generated using the force-directed layout algorithm provided by Cytoscape, which puts nodes with the highest number of neighbors closer to the center of the graph. The reader will notice that the network graphs become denser from S1 to S2 and from S2 to S3, indicating that the know-well connections increased after design changes were made to the website and increased again when the third survey was administered.

From a quantitative perspective, there was an increase in the average number of neighbors, number of edges, network density, and multiedge node pairs each from S1 to S2 and from S2 to S3. The increase in the average number of neighbors was 1.28 from S1 to S2 and 0.71 from S2 to S3. Similarly, the increase in the number of edges was 16 from S1 to S2 and 12 from S2 to S3. This means there were 16 more know-well connections in S2 from S1 and 12 more know-well connections in S3 from S2. This effect also occurred for the network density, increasing 0.08 from S1 to S2 and 0.07 from S2 to S3. These results suggest that the effect in the increase of connections was more pronounced immediately after the redesign of the website, a somewhat expected effect of implementing the new site.

Multiedge node pairs, the know-well connections that went both ways between pairs of respondents, did not follow the same pattern but rather increased throughout the study. Multiedge node pairs increased by 5 from S1 to S2 and by 7 from S2 to S3. This result could suggest a longer term development of strong reciprocal relationships.

In Figure 8, the visual structures of the difference graphs were also generated by the force-directed layout algorithm in Cytoscape. The S1-S2 difference graph shows all the edges in S2 that did not appear in S1. Similarly, the difference graphs of S2-S3 and S1-S3 show new edges that appeared from S2 to S3 and S1 to S3, respectively. It is important to note that in the difference graphs, the number of edges does not correspond with a direct difference of the edges at each survey in Figure 7. This is because some respondents may have considered another member as a know-well connection in one survey but not in subsequent surveys. For example, respondent 612 claimed to know 510 well in S1 but not in S2. The difference graphs show only new edges and not where edges may have been lost. Although this may not be ideal from a quantitative perspective, the difference graphs were considered useful in analyzing the changes from survey to survey because they are less dense than the individual survey graphs in Figure 7.

The actor-organization difference graphs, in Figure 8, show more new connections in S1-S2 than in S2-S3, but in each graph, these connections seemed to be uniformly distributed over the total number of respondents. In S1-S2, the number of new connections was 23, whereas the number of new connections in S2-S3 was 17. Although there was a decrease in the number of new connections from S1-S2 to S2-S3, the new connections were distributed over 93% of the total respondents (13 of 14) in S1-S2 and 79% of the respondents in S2-S3 (11 of 14). Correspondingly, in S1-S3, the new connections were distributed over 86% of the population (12 of 14). Overall, the S1-S2, S2-S3, and S1-S3 difference graphs each show a fairly uniform distribution to the increase in edges, suggesting more uniform effects on the UCP-SARnet community of practice. The uniform distribution of new connections in the network is showing a broad change that affected most respondents.

For further analysis, Table 2 shows the mean number of know-well connections over the 14 base respondents at S1, S2, and S3. In agreement with the SNA graphs, the mean number of connections increased more between S1 and S2 than between S2 and S3. The standard error for each survey was high, showing some overlap between surveys, indicating that there was a range of numbers of connections in which some respondents knew many other respondents well, whereas others knew relatively fewer respondents well. A repeated measures ANOVA, with a Greenhouse-Geisser correction, determined that the mean number of know-well connections differed statistically significantly from S1, S2, and S3; F(1.159, 15.064) = 8.361, p < .05. A post hoc test revealed that the mean increase of five connections from S1 to S2 was significant (p < .05). Furthermore, the mean increase of 2.42 between S2 and S3 was significant (p < .05). This suggests that there was an effect of the timing of the surveys: S1, before the design changes; S2, about 2 months after the design changes; and S3, about 5 months after the design changes.

Table 2:

Mean Number of Know-Well Connections at the Three Survey Points

	Survey 1	Survey 2	Survey 3
Mean number of know-well connections	8.33	13.33	15.75
Standard error	2.03	2.98	3.27

Actor-Organization Social Network Analysis

Figure 9 shows the actor-organization SNA for the 14 base respondents of the UCP-SARnet leadership team at S1, S2, and S3. These graphs show the strong connection between respondents and the organizational partners of UCP-SARnet at the time of each survey and are accompanied with statistics for comparison. Similar to the actor-actor results in Figure 8, the graphs in Figure 10 present the new strong connections created between S1 and S2, S2 and S3, and S1 and S3.

Figure 9.

Actor-organization connections. The circle nodes contain random numbers that correspond to the 14 base survey respondents, whereas the white squares represent organizational partners.

Figure 10.

Actor-organization difference graphs. The circle nodes contain random numbers that correspond to the 14 base survey respondents, whereas the white squares represent organizational partners.

In Figure 9, the visual structure of each graph was again generated using the force-directed layout algorithm provided by Cytoscape. These graphs became denser from S1 to S2 and from S2 to S3, suggesting that there was an increase in strong connections after the website changes were made and after S2 was administered.

From a quantitative perspective, there was an increase in the average number of neighbors, number of edges, and number of nodes from S1 to S2 and S2 to S3. The difference of the average number of neighbors was 0.63 from S1 to S2 and increased to 0.87 from S2 to S3. The number of nodes from S1 to S2 increased by 10, whereas the number of nodes from S2 to S3 increased by only 2. Similarly, the number of edges from S1 to S2 increased by 38, whereas the number of edges in S2 to S3 increased by 28. The density of the network is based on the number of possible connections, and the increase in edges was not enough to offset the increase in nodes from survey to survey. As a result, the network density remained stable at about 0.10 to 0.11 through each survey point. Nonetheless, although the density did not increase, these results suggest the actor-organization network growth, in terms of number of nodes occurred and the growth, was more pronounced from S1 to S2 than from S2 to S3.

In Figure 10, the visual structures of the difference graphs were also generated by the force-directed layout algorithm in Cytoscape. Like the difference graphs of the actor-actor SNA, the S1-S2 difference graph shows all the edges in S2 that did not appear in S1, and the S2-S3 and S1-S3 difference graph shows all the new edges that appeared from S2 to S3 and from S1 to S3, respectively. Again, it is important to note that the difference graphs do not correspond with a direct difference of edges from survey to survey, as shown in Figure 9, because some respondents may have considered an organizational partner a strong connection in one survey but not in subsequent surveys—some edges may have been lost from survey to survey. Again, the difference graphs are helpful to see the nature of the changes from survey to survey because they are less dense than the survey graphs in Figure 9.

The actor-organization difference graphs, in Figure 10, show an increase in the number of connections between each survey, with one or two of the respondents making many new strong connections. In S1-S2, the number of new connections was 92, whereas the number of new connections in S2-S3 was 82. S1-S2 shows that new connections were distributed over 71% of the base respondents (10 of 14). Similarly, S2-S3 and S1-S3 show that new connections were distributed over 79% of the base respondents (11 of 14) and 71% of the base respondents (10 of 14), respectively. However, in contrast to the actor-actor difference graphs in Figure 4, many of the new connections were centered on just a few of the actors. In S1-S2, many of the new connections were centered on Actor 194 and Actor 090, and in S2-S3, many of the connections were centered on Actor 585. What the reader will notice is that in the S1-S3 graph, Actor 194 shows only one new connection, which suggests that this respondent may have misinterpreted the survey question. Regardless, the actor-organization difference graphs show different characteristics than the actor-actor graphs, and the new connections were not uniformly distributed over the respondents, suggesting that isolated factors may have affected some of the respondents individually.

For further statistical analysis, Table 3 shows the mean number of strong connections over the 14 base respondents at S1, S2, and S3. In agreement with the SNA graphs in Figure 9 and Figure 10, the mean number of connections increased more between S1 and S2 than between S2 and S3. Similar to the actor-actor SNA, the standard error for each survey was high, showing overlap between surveys, indicating that there was a range in the number of strong connections, where some respondents said they knew many organizations well, whereas others knew relatively fewer organizations well. A repeated measures ANOVA determined that the mean number of strong connections did not differ in a statistically significant way from S1, S2, and S3; F(2, 22) = 1.450, p = .256.

Table 3:

Mean Number of Strong Actor-Organization Connections

	Survey 1	Survey 2	Survey 3
Mean number of strong connections	6.43	9.14	11.14
Standard error	1.84	2.48	3.25

Website Communication

The website communication analysis consisted of statistics based on the number of content posts to the UCP-SARnet site over each period, particularly the number of blog posts and comments, as well as unique bloggers and commenters each week.

Figure 11 shows the content statistics based on the number of posts to UCP-SARnet over P1, P2, and P3. Overall, a strong increase was seen in P2, followed by a strong decrease by P3.

Figure 11.

Mean weekly content posts based on post type on the UCP-SARnet web portal over Period 1, Period 2, and Period 3.

Although the data did not meet the homeoscedasticity assumptions for MANOVA, they were amenable to examination by ANOVA. Univariate ANOVA tests were conducted for each of the website content analysis variables. The univariate ANOVAs showed a significant effect over the three periods on weekly blog posts (F(2, 27) = 16.630, p < .05, partial ε² = .552), weekly comment posts (F(2, 27) = 5.747, p < .05, partial ε² = .299), and weekly unique commenters and bloggers (F(2, 27) = 20.247, p < .05, partial ε² = .600).

Post hoc tests between periods were conducted using the Tukey honestly significant difference (HSD) test. Mean scores for weekly blog posts were different between P1 and P2 (p < .005), P2 and P3 (p < .05), and P1 and P3 (p < .05). Mean scores for weekly comments were statistically different between P1 and P2 (p = .05) and P2 and P3 (p < .05) but were not statistically different between P1 and P3 (p < .960). Finally, mean scores for weekly unique commenters and bloggers were statistically different between P1 and P2 (p < .05), P2 and P3 (p < .05), and P1 and P3 (p < .05). In general, these results supported the idea that, over time, website content had increased, although there was a noticeable transient increase as a result of the release of the new site.

Discussion

Overall, there were several interesting results from this study. First, by adding insights from the principles for supporting CoPs to an existing human factors method, the work domain analysis portion of CWA yielded a richer analysis with community-oriented elements. This analysis was used to generate design ideas. On evaluation, there is evidence that the new design achieved some of the goals identified in the analysis—in particular, increasing strong connections between people and external organizations, as well as encouraging communication through the website.

From the evaluation, we were able to show significant increases in connections between members of UCP-SARnet following the redesign of the website. Although these connections increased rapidly after the new site was launched, they continued to increase throughout the evaluation period. In particular, know-well connections increased, suggesting that the new site was effective at increasing connections between the members of the community.

Although qualitative increases were seen in actor-organization connections, these changes were not statistically significant. One observation from the data suggests that individual actors may build connections at very different rates, with some building many new connections and others building fewer connections. This kind of information cannot be captured well through a measure such as SNA, and some interesting individual differences may be hidden.

In examining the quantity of website communication, we observed a strong increase in communication quantity following the launch of the new website. Although communication levels dropped off at the end of the final period, there was still evidence of more communication than before the new site had been launched. It is reasonable to expect that some of the increased strength in connections between members and organizations could have resulted from increased communication facilitated by the website.

Some limitations need to be considered in a study of this nature. First, the community under study may be naturally evolving over time, so although it is tempting to attribute the changes we observed to the new website we designed, other factors could have been involved. From personal communications with the organization leaders after the evaluation, we believe this effect to have been minimal. They reported no organizational changes made during the year after the website was introduced and stated that “anything you have observed in the year 2011 was a result of modification of the website” (M. Wosinski, personal communication, November 5, 2012). We feel that the approach taken here, examining one community over time, provides insights that the approach may have had an effect, as well as suggestions of a promising approach to be considered in other designs. This approach is similar to other longitudinal studies of website use (e.g., Lee, 2012; Leslie, Marshall, Owen, & Bauman, 2005; Stevens et al., 2008). To further develop the design approach in future work, we would create specific hypotheses about the effects of applying different types of design changes to the website, apply these changes incrementally, and carefully track the effects on website usage, website communications, and social network graphs. With this approach, we would try to link the different types of design changes to factors we model in the community through the effects of the design changes. Second, although the quantitative analysis shows that the improvements to the website had an effect on connections and amount of communication, there is a need to link the quantitative results to qualitative observations to help explain what was changing in the social network. For example, we would like to answer the following questions: What was the nature of the increase or loss in know-well connections for any individual actor? What kinds of interactions between UCP-SARnet members may have given rise to changes in the network structure? Some qualitative techniques could be used to fortify the SNA, such as conducting a narrative inquiry (Clandinin & Connelly, 2000) or a more traditional content analysis (Krippendorff, 1980). For this research in particular, it may be helpful to conduct a content analysis using the website blog posts and comments and categorize the interactions as they relate to the practices listed in Figure 1 to see if there was a change from survey period to survey period. Those changes may provide information to help explain increased or decreased connections in the network. With respect to a narrative inquiry, it may be possible to conduct interviews or focus groups where the members themselves see the network maps and talk about what may have happened. For example, perhaps built connections were the result of a strategy they employed in using the website or the result of a particular event.

Third, we cannot be certain of the generalizability of this work. This particular community was a very socially active community and may have responded better to these interventions than other communities. This community had existing strong values that likely influenced their behavior. In addition, the survey respondents who continued throughout the course of a several-month study may have been more motivated than average community members. To consider how different communities may respond differently to design interventions such as the one in this study, some researchers have created a typology of virtual CoPs to help to describe the “unique personality” of each CoP using a set of structural characteristics (Dubé, Bourhis, & Jacob, 2006). Use of such a typology could provide a frame for comparing the effects of interventions on CoPs with similar structural characteristics.

We consider these limitations to present challenges for further research. With more studies, social design interventions may become better understood. A stronger base of research in this area will be important for human factors as a field in a socially networked world.

Conclusion

There are several key lessons from this work. First, existing human factors methods can be adapted to suit the concerns of highly social environments. We have shown that an adaptation is possible and that by combining human factors methods with other approaches—in this case, CoPs and CWA—effective designs can result. The CoPs approach, although not developed for systems design, provided a view of community growth objectives and practices that were useful for designing a new system. Human factors methods, such as CWA, provide strong foundations for developing effective designs to improve human performance and are well poised to take advantage of relevant concepts from other fields such as social psychology. We believe that using the CoPs framework generated new insights into the work domain and allowed us to develop a richer analysis of the problem space than would have been possible using CWA in its classical form. In this way, this work contributes a richer perspective when using CWA. Using social network analysis as an evaluation approach, we observed significant changes in measures relevant to assessing the connectivity of people in the CoP we were observing. For human factors research to stay relevant in socially networked environments, researchers and practitioners will need to make use of social psychology, the CoPs framework, and social network analysis.

Key Points

The communities of practice concept was used to inspire an existing human factors method, cognitive work analysis, to generate community relevant design requirements.

In a longitudinal study, where a new CoPs-CWA design was compared with the previous baseline design, the new design showed increases in social connections between people and organizations, as well as increased communications, when studied over time.

Footnotes

We are thankful to the members of the UCP-SARnet leadership team for participating in this study. We also acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC).

Adam Euerby received an MASc in systems design engineering from the University of Waterloo in 2012. His research draws on engineering,psychology,philosophy,and cognitive science to develop new ways of supporting enterprise communities using social networking technology. He is currently a product designer at Desire2Learn in Kitchener,Canada.

Catherine M. Burns received a PhD in mechanical and industrial engineering from the University of Toronto in 1992. She is a professor of systems design engineering at the University of Waterloo,Canada,and directs the Advanced Interface Design Lab,where she explores human factors approaches to interface design for complex systems. She is known for her work in cognitive work analysis and ecological interface design.

References

AllBusiness. (2012, December 7). How often should you redesign your company’s web site [Web log message]? Retrieved from www.allbusiness.com/technology/internet-web-design/1211-1.html

Ashoori

Burns

C. M.

(2013). Team cognitive work analysis: Structure and control tasks. Journal of Cognitive Engineering and Decision Making, 7(2), 123-140. doi:10.1177/1555343412445577

Belbin

R. M.

(1981). Management teams: Why they succeed or fail. Oxford, MA: Butterworth-Heinemann.

Christiansen

(2005, April). Boundary objects, please rise! On the role of boundary objects in distributed collaboration and how to design for them. Paper presented at the 2005 Workshop of Cognition and Collaboration, Portland, OR.

Clandinin

D. J.

Connelly

F. M.

(2000). Narrative inquiry: Experience and story in qualitative research. San Francisco, CA: Jossey-Bass.

Cooke

N. J.

(2005). Measuring team knowledge. In Stanton

Hedge

Brookhuis

Salas

(Eds.), Handbook on human factors and ergonomics methods. Boca Raton, FL: CRC Press.

Crandall

Klein

Hoffman

R. R.

(2006). Working minds: A practitioner’s guide to cognitive task analysis. Boston, MA: Bradford Books.

Cuevas

H. M.

Fiore

S. M.

Salas

Bowers

C. A.

(2002). A macroergonomic approach to distributed team performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46, 1335-1339.

Dubé

Bourhis

Jacob

(2006). Towards a typology of virtual communities of practice. Management, 1(2), 69-93.

10.

Endsley

M. R.

(2011). Designing for situation awareness: An approach to user-centered design (2nd ed.). Boca Raton, FL: CRC Press.

11.

Endsley

M. R.

Bolte

Jones

D. G.

(2003). Designing for situation awareness: An approach to user-centered design. Boca Raton, FL: Taylor & Francis.

12.

Euerby

Burns

C. M.

(2012). Designing for social engagement in online social networks using community of practice theory and cognitive work analysis: A case study. Journal of Cognitive Engineering and Decision Making, 6(2), 194-213.

13.

Hendrick

Kleiner

(2002). Macroergonomics: Theory, methods and applications. Boca Raton, FL: CRC Press.

14.

IGLOO Software. (2011). Retrieved March 25, 2011, from http://www.igloosoftware.com/

15.

Klein

(2000). Cognitive task analysis of teams. In Schraagen

J. M.

Chipman

S. F.

Shalin

V. L.

(Eds.), Cognitive task analysis. Mahwah, NJ: Lawrence Erlbaum.

16.

Krippendorff

(1980). Content analysis: An introduction to its methodology. Newbury Park, CA: Sage.

17.

Lee

(2012). An elaboration likelihood model based longitudinal analysis of attitude change during the process of IT acceptance via education program. Behaviour and Information Technology, 31, 1161-1171.

18.

Leslie

Marshall

A. L.

Owen

Bauman

(2005). Engagement and retention of participants in a physical activity website. Preventative Medicine, 40, 54-59.

19.

Malone

T. W.

Crowston

(1990). What is coordination theory and how can it help design cooperative work systems? Proceedings of the 1990 ACM Conference on Computer-Supported Cooperative Work, 357-370.

20.

Naikar

Drumm

Pearce

Sanderson

M. P.

(2000). Designing new teams with cognitive work analysis. In Proceedings of the Fifth Australian Aviation Psychology Symposium. Manly, Australia.

21.

Pfautz

J. D.

Pfautz

S. L.

(2009). Methods for the analysis of social and organizational aspects of the work domain. In Bizantz

A. M.

Burns

C. M.

(Eds.), Applications of cognitive work analysis. Boca Raton, FL: CRC Press.

22.

Salas

Dickinson

T. L.

Converse

S. A.

Tannenbaum

S. I.

(1992). Toward an understanding of team performance and training. In Swezey

R. W.

Salas

(Eds.), Teams: Their training and performance (pp. 3-31). Norwood, NJ: Ablex.

23.

Scott

(1992). Social network analysis. Newbury Park, CA: Sage.

24.

Stevens

V. J.

Funk

K. L.

Brantley

P. J.

Erlinger

T. P.

Myers

V. H.

Champagne

C. M.

Bauck

Samuel-Hodge

C. D.

Hollis

J. F.

(2008). Design and implementation of an interactive website to support long-term maintenance of weight loss. Journal of Medical Internet Research, 10(1), e1.

25.

Tuckman

B. M.

(1965). Development sequence in small groups. Psychological Bulletin, 63(6), 384-399.

26.

UCP-SARnet. (2012). UCP-SARnet home. Retrieved March 13, 2011, from http://www.ucpsarnet.org

27.

Vicente

K. J.

(1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. Mahwah, NJ: Lawrence Erlbaum.

28.

Wenger

(1998). Communities of practice: Learning, meaning, and identity. Cambridge, UK: Cambridge University Press.

29.

Wenger

McDermott

Snyder

(2002). Cultivating communities of practice: A guide to managing knowledge. Boston, MA: Harvard Business School Press.