Sage Journals: Discover world-class research

Abstract

Objectives:

To develop an objective, structured observational tool to enable identification and measurement of hazards in the built environment when applied to audiovisual recordings of simulations by trained raters.

Background:

Simulation-based facility design testing is increasingly used to optimize safety of healthcare environments, often relying on participant debriefing or direct observation by human factors experts.

Methods:

Hazard categories were defined through participant debriefing and detailed review of pediatric intensive care unit in situ simulation videos. Categories were refined and operational definitions developed through iterative coding and review. Hazard detection was optimized through the use of structured coding protocols and optimized camera angles.

Results:

Six hazard categories were defined: (1) slip/trip/fall/injury risk, impaired access to (2) patient or (3) equipment, (4) obstructed path, (5) poor visibility, and (6) infection risk. Analysis of paired and individual coding demonstrated strong overall reliability (0.89 and 0.85, Gwet’s AC₁). Reliability coefficients for each hazard category were >0.8 for all except obstructed path (0.76) for paired raters. Among individual raters, reliability coefficients were >0.8, except for slip/trip/fall/injury risk (0.68) and impaired access to equipment (0.77).

Conclusions:

Hazard Assessment and Remediation Tool (HART) provides a framework to identify and quantify hazards in the built environment. The tool is highly reliable when applied to direct video review of simulations by either paired raters or trained single clinical raters. Subsequent work will (1) assess the tool’s ability to discriminate between rooms with different physical attributes, (2) develop strategies to apply HART to improve facility design, and (3) assess transferability to non-ICU acute care environments.

Keywords

simulation patient safety research instruments tools surveys patient room design staff safety PICU

Introduction

Healthcare facility design may impact patient and staff safety by introducing unanticipated latent conditions in the clinical workspace (Birnbach et al., 2010; Gignon et al., 2017; Nickson et al., 2021; Reiling, 2006). While simulation-based facility design testing (SBDT) is increasingly utilized to identify latent conditions prior to construction, development and execution of these efforts vary widely, heavily influenced by leadership support, receptivity of the architecture team to simulation findings, availability of simulation and human factors expertise, staffing constraints, and construction time lines. In this context, there is a need for reliable tools to allow simulationists to generate specific, actionable findings directly related to the physical environment. Current methodology relies heavily on participant debriefing and surveys or direct observation or video review by human factors experts. The first approach captures only individual clinician experience, is subject to recall bias, and cannot reliably quantify relative frequency of hazards, while the second approach demands significant specific human resources, potentially limiting feasibility and scalability of SBDT. In contrast, post hoc review of recorded simulations using a structured observational tool explicitly focused on the physical workspace and applicable by coders with clinical and simulation backgrounds may facilitate objective identification of workplace hazards. Accompanying descriptive data on such hazards could be leveraged to adapt facility design to mitigate risk. Quantification of relative frequency of various hazard types could be used to assess changes in hazard profiles via repeat simulation following design changes.

...post hoc review of recorded simulations using a structured observational tool explicitly focused on the physical workspace and applicable by coders with clinical and simulation backgrounds may facilitate objective identification of workplace hazards. Accompanying descriptive data on such hazards could be leveraged to adapt facility design to mitigate risk.

To successfully identify latent threats prior to construction, the design team must understand the dynamic ways in which patients and healthcare staff interact with the built environment (Carayon et al., 2006; Colman, Dalpiaz, et al., 2020; Kaba & Barnes, 2019). Traditional architectural blueprints may be difficult for clinical teams to interpret and do not support functional evaluation to identify latent conditions (Adler et al., 2018; Bayramzadeh et al., 2018; Colman et al., 2019b; Reiling, 2006; Taylor et al., 2015; Wingler et al., 2019). While virtual reality platforms and scale mock-ups may help clinical teams visualize design, these tools, in isolation, do not enable functional analysis (Wingler et al., 2019). Hence, preconstruction simulation-based testing has emerged as a powerful tool to identify design elements that may negatively impact safety, potentially avoiding costly postconstruction changes or risk to patients and staff (Broberg et al., 2011; Colman, Edmond, et al., 2020; Garmer et al., 2004; Taylor et al., 2014). Conducting simulation in scale mock-ups allows teams to evaluate “work as done” as opposed to “work as imagined” in a proposed space (Deutsch, 2017).

Conducting simulation in scale mock-ups allows teams to evaluate “work as done” as opposed to “work as imagined” in a proposed space

Video-recorded simulations allow repeated review of scenarios, mitigating recall bias and allowing an assessment of hazard frequency (Blike et al., 2005; Fan et al., 2016; Geis et al., 2011; Oakley et al., 2006). For example, video review by human factors experts has been leveraged to broadly identify hazard themes related to the physical workspace during in situ simulations in an existing trauma bay (Petrosoniak et al., 2021). While this approach advances the rigor with which direct observation can yield quantitative, actionable data related to workspace hazards, reliance on human factors experts may limit generalizability. In the absence of specific human factors expertise, others have relied on application of existing safety frameworks, such as evidence based Safe Design Principles, to aid in hazard identification (Colman, Edmond, et al., 2020; Joseph et al., 2012). This set of 11 principles provides broad goals such as “minimize environmental hazards” or “reduce risk of injury.” Similarly, the Systems Engineering Initiative for Patient Safety (SEIPS) Model 2.0 (Holden et al., 2013) provides five key themes to be considered in systems evaluation: “person, tools and technology, tasks, organization, and environment.” Because the themes generated through human factors observations and both the Safe Design Principles and SEIPS frameworks lack rigorous definitions to guide event detection, utilization of this approach by clinicians and simulationists may be challenging. Thus, we set out to develop an objective, quantitative tool to aid identification and measure frequency of potential hazards in the physical workspace. In order to ensure applicability by those with diverse skillsets and bolster identification of potential safety threats, specific hazard categories and associated operational definitions were developed to guide observation of recorded simulations.

The goals in development of the tool were to (1) aid reliable, reproducible identification of workplace hazards in the clinical environment; (2) provide a quantitative assessment of overall hazard burden to allow measurement over time with successive design iterations; and (3) enable collection of descriptive data for each identified hazard to aid mitigation. This article describes the development of the tool and establishment of interrater reliability (IRR).

Method

Overview of Study Design

Figure 1 summarizes the process used for development and assessment of reliability of the Hazard Assessment and Remediation Tool (HART). Simulation characteristics are summarized in Table 1. Participants in research simulations provided informed consent, and this study was approved by the institutional review board.

Figure 1.

This figure provides a schematic overview of the process by which Hazard Assessment and Remediation Tool was developed and outlines how interrater reliability of the tool was established. The figure shows key outputs at each phase of the study.

Table 1.

Characteristics of Simulations and Audio–Video Recordings Used for Development and Establishment of Interrater Reliability of the Hazard Assessment and Remediation Tool.

Video #	Room Type	Standardized Room Setup (Y/N)	Simulation Purpose	Scenario	# Participants	Minutes Reviewed
Hazard category development/refinement
1	Mock-up	N	Design evaluation	ECPR	13	23
2	Mock-up	N	Design evaluation	Cardiac arrest	13
3	In situ ICU	N	Education	Cardiac arrest	10	21
4	In situ ICU	N	Education	Cardiac arrest	10	23
5	On Unit Sim Lab	N	Education	ECPR	12	2
6	In situ ICU	N	Education	ECPR	12	25
Operational definition development/refinement
7	In situ ICU	N	Education	ECPR	12	25
8	In situ ICU	N	Education	ECPR	13
9	On Unit SIM Lab	N	Education	Airway emergency	10	9
6	In situ ICU	N	Education	ECPR	12	2
10	In situ ICU	N	Education	ECMO circuit emergency	12	6
5	On Unit Sim Lab	N	Education	ECPR	12	6
11	On Unit Sim Lab	N	Education	ECPR	11	5
12	On Unit Sim Lab	N	Education	ECPR	13	3
Determination of sources of rater discrepancy; refinement coding protocols, A-V capture
13	On Unit Sim Lab	N	Education	ECPR	13	3
14	On Unit Sim Lab	N	Education	ECPR	11	6
15	ICU Sim Lab	Y	Research	Cardiac arrest	8	9
16	ICU Sim Lab	Y	Research	Cardiac arrest	8	6
17	ICU Sim Lab	Y	Research	Cardiac arrest	9	3
Novice clinical rater training
11	On Unit Sim Lab	N	Education	ECPR	11	7
18	ICU Sim Lab	Y	Research	Cardiac arrest	8	2
15	ICU Sim Lab	Y	Research	Cardiac arrest	8	2
Assessment of interrater reliability
19	ICU Sim Lab	Y	Research	Cardiac arrest	9	17
20	ICU Sim Lab	Y	Research	Cardiac arrest	8	9
21	ICU Sim Lab	Y	Research	Cardiac arrest	8	8

Note. Y/N: yes/no; ECMO: extracorporeal membrane oxygenation; ECPR: extracorporeal cardiopulmonary resuscitation; ICU: intensive care unit.

Development of HART Hazard Categories and Operational Definitions

An iterative approach was used to identify and refine hazard categories related to the physical workspace. First, qualitative data from participant debriefings of pediatric intensive care unit (PICU) simulations were used to generate hazard themes. In order to evaluate proposed design prior to construction of a new clinical tower at a children’s hospital, interdisciplinary teams participated in simulations of crisis events in mock-ups of proposed PICU and neonatal intensive care unit (NICU) rooms. Full scale mock-ups, constructed of drywall, included key elements of room design and construction including actual patient care booms, outlets, medical gas ports, windows, and doors. Realism of simulations was optimized through the engagement of a full team of multidisciplinary providers, use of real clinical equipment, simulation scenarios of high relevance to the teams involved, and high-fidelity human patient simulators. Prior to simulation, teams were briefed that the objective of the simulations was to evaluate elements including but not limited to room size, location of doors and windows, placement of fixed equipment, and ease of working in the room. Following simulation but prior to group debriefing, participants recorded observations on any elements of the physical environment that would impede safe, timely, and effective care delivery or risk injury to staff or patients. Additional hazards were surfaced during group debriefing of participants and observers. Observers included individuals from the hospital facilities team, architects, building project leadership, and environmental health and safety.

Observations from participants and observers were analyzed using the approach of Braun and Clarke (Braun, 2006). Thematic analysis of 55 unique participant and observer reflections yielded eight hazard categories. Next, the research team conducted successive cycles of group video review to identify hazards in videos of resuscitation simulations conducted either for preconstruction design testing (drywall mock-ups) or simulation-based team training (in situ simulations) in existing clinical environments. Hazards were assigned to the previously identified categories in order to validate accuracy and comprehensiveness of categories. This broader simulation sampling strategy allowed for incorporation of ICU rooms of varied sizes and spatial configurations and clinical scenarios with different equipment and team composition requirements (cardiac arrest, airway emergency, and extracorporeal membrane oxygenation cannulation during CPR), and different patients sizes.

Subsequently, the team drafted operational definitions for each hazard category describing common, observable manifestations of hazards to allow trained observers to objectively identify and categorize hazards using the tool. In order to ensure greater breadth in identification of hazards, the research team included those with clinical expertise: critical care physician (C.A.) as well as members with specific background in industrial engineering (L.D.), environmental health and industrial hygiene (H.B.), and process improvement (S.A., K.G.).

Operational definitions were further refined through iterative cycles of individual coding followed by team video review and discussion. For each hazard coded by an individual, the group reached a consensus on whether the instance should be considered a hazard based on operational definitions as documented; operational definitions were then revised to reduce ambiguity based on these discussions. Further input on hazard categories and operational definitions was sought and incorporated from subject matter experts in intensive care, emergency medicine, surgery, simulation, human factors, and facility design testing.

Refinement of HART and Establishment of Coding Protocols

Preliminary assessment of HART showed modest agreement between raters. In response, a detailed line-by-line review of coding exercises was conducted, identifying causes of discrepancies for each instance of rater nonagreement. Themes summarizing sources of variance were generated; the most frequent causes included variable event detection, overlap in hazard categories, gaps in clinical context for nonclinical raters, and lack of clarity in operational definitions (Figure 2). These insights were used to revise operational definitions to enhance specificity, update coding practices to support event detection, and improve audiovisual capture practices.

Figure 2.

Sources of Rater Discrepancies.

In order to improve event detection, audio and video capture were optimized through four pilot simulations prior to conducting formal assessment of IRR. Final video capture utilized two wide angle cameras; one from the foot of the bed showed unobstructed views of patient right and left and the patient bed. The second camera, positioned near the ceiling to the patient’s left midway from head to foot of the bed yielded unobstructed views of head and foot of bed (including code cart), and patient right. Three independent microphones were used to capture distinct conversations, including one at the foot of the bed, one on patient left boom to capture conversations at the airway, and one on the code cart. Figure 3 illustrates the standardized setup that is used to show optimal positioning of video cameras and microphones in the Intensive Care Unit setting. All video cameras and 2 microphones are in fixed locations during the simulations. A third microphone, affixed to the code cart, enables capture of key audio data related to code cart activities regardless of positioning within the room.

Figure 3.

Standardized Room Set-up for Simulations used for Establishment of Interrater Reliability.

Structured coding protocols were introduced to enhance event capture. Video was segmented into 10-s increments for sequential review from each camera angle to optimize coder attention and reduce fatigue (Fraser et al., 2015). The full video was reviewed for one hazard category at a time. Thus, each 10 second increment was reviewed 12 times—once from each of two camera angles for each of the six hazard categories. Mangold Interact software (Mangold, 2020) was used to segment video, conduct video review and coding, and record descriptive data for each hazard for subsequent qualitative analysis. Qualitative data collected included role of individual impacted by the hazard, equipment or room feature contributing to hazard, location in room where hazard noted, and general description of hazard. While detailed process for analyzing qualitative data is outside the scope of this article, Figure 4 provides a summary of the overall process for application of HART to a specific facility design testing or improvement project.

Figure 4.

This figure outlines the process by which Hazard Assessment and Remediation Tool can be applied to collect data for a facility improvement project, including analysis of both quantitative and qualitative data to guide improvements.

Evaluation of sources of coding variance also identified opportunities to provide additional training on clinical context to support non-clinical raters, particularly related to infection and visibility hazards. For infection hazards, non-clinical raters struggled to identify which breaches were related to room design versus those that represented failure to follow best practice. To address this challenge, additional specificity was added to the operational definitions; non-clinical coders were trained in infection prevention fundamentals, detailed infection risk and visibility reference guides were documented; and a scenario guide was created, identifying simulation participants by clinical role and outlining associated responsibilities (see Supplemental Digital Content A).

In order to ensure interpretation of clinical context and to further improve event detection, paired coding was introduced. Each coding pair, composed of one clinical and one non-clinical coder, reviewed and coded videos together, allowing clinical coders to provide clinical context and non-clinical coders to challenge clinician assumptions around the inevitability of commonplace hazards. Formal assessment of IRR was conducted between coding pairs. To test deployment of HART by individual coders with clinical and simulation backgrounds, IRR between coders with background in ICU practice (C.A., G.U.) and simulation (C.A.) was also assessed.

Assessment of IRR—Statistical Methods

Formal assessment of IRR was undertaken for both pair and single-rater approaches using videos of two PICU simulations with a standardized simulation delivery approach and optimized camera angles. Scenarios required an interdisciplinary team response to cardiac arrest due to sepsis or myocarditis; segments were sampled from the onset of cardiac arrest in order to ensure sufficient density of hazards. Raters were provided with a scenario guide detailing the clinical context of the scenario and roles and responsibilities of clinicians involved.

IRR of the HART tool was determined using Gwet’s First-Order Agreement Coefficient. The tool, scored independently by raters blinded to the other’s scores, produces binary ratings. Due to high agreement and low trait prevalence (occurrence of hazards), Gwet’s AC₁ was selected as the appropriate statistic, as it mitigates the challenges associated with Cohen’s κ when there is significant imbalance between prevalence of binary ratings (Feinstein & Cicchetti, 1990; Wongpakaran et al., 2013). Subjects, defined as 10-s segments of video, were considered to be independent. If multiple unique events in the same hazard category were detected in a single 10-s segment, they were also assumed to be independent.

Calculations were performed using Agreestat software (https://www.agreestat.com) and are presented alongside Percent Agreement and Cohen’s κ. Sample size was calculated based on the methodology described by Gwet using a 15% margin of error, resulting in a sample size of 48 segments (Gwet, 2014). For hazard categories in which more than one hazard was observed during a 10-s increment, greater than 48 subjects were analyzed (Table 2).

Table 2.

Count of Video Segments Analyzed.

Hazard Categories	Paired Coding		Individual Coding
Hazard Categories	Actual Video Segments	Expected Video Segments	Actual Video Segments	Expected Video Segments
Slip, trip, fall, or injury risks	49	48	53	48
Obstructed access to equipment/ or supplies	48	48	48	48
Obstructed access to patient	49	48	48	48
Obstructed pathway	52	48	49	48
Obstructed visibility	49	48	48	48
Infection risk	68	66	48	48
Total	135	306	294	288

Note. Table 1 details the number of subjects analyzed for each hazard category and for the tool overall during the establishment of interrater reliability among paired clinical/nonclinical raters and individual clinical coders. Each video segment is 10s in duration; number of segments > expected occurs when multiple hazards of the same hazard category occur within a single 10-s segment of video.

Results

Hazard Categories and Operational Definitions

Study conduct and reporting adhered to the GRASS guidelines (Kottner et al., 2011). Initial thematic analysis of participant debriefings identified eight potential hazard categories. Following iterative review of recorded simulations by the research team, six hazard categories were finalized, including (1) slip, trip, fall, or injury risk; (2) obstructed access to patient; (3) obstructed access to equipment; (4) obstructed pathway (for equipment or people); (5) infection risk; or (6) poor visibility. Poor ergonomics was initially considered as a hazard category but was eliminated due to challenges in achieving a sufficiently objective operational definition that could be applied without specific expertise in ergonomic assessment and did not add substantially to the coding burden. Limited ergonomic assessment was incorporated into the hazard categories “obstructed access to patient” and “obstructed access to equipment,” with awkward positioning undertaken to access the patient or key equipment specifically included in operational definitions. These additions were designed to capture characteristics of the physical space that required workers to engage in repeated bending, twisting, or reaching that over time contribute to risk of employee injury (Brogmus et al., 2007; Kroemer, 1989; Rogers et al., 2013).

Operational definitions for each of the six categories were iteratively refined, resulting in the final set of operational definitions summarized in Table 3 (complete HART instrument available in Supplemental Digital Content B).

Table 3.

Hazard Assessment and Remediation Tool: Summary of Hazard Categories and Brief Operational Definitions.

Hazard Category	Description	Operational Definition
Slip, trip, fall, or injury risks^a	Clinicians experience a risk of slipping, tripping, falling, or injury in the clinical environment. For instance, this type of hazard is frequently observed in relation to cords stretched across the clinical space.	Latent conditions Person ducks under equipment or cord Person steps on or over equipment or cord Slip or trip without fall, injury, or disruption Person steps over item or spill Person kicks item on the floor Person disturbs cord with their foot/leg Other
Slip, trip, fall, or injury risks^a		Active failures Slip or trip with injury or fall Person’s task is disrupted due to slip, trip, or injury risk, leading to delay in care Slip, trip, or fall leads to compromise or disruption of equipment Injury due to contact with equipment, people, or environment Other
Obstructed access to equipment or supplies^a,c,f,g	Access to equipment is obstructed by immediately adjacent people or equipment, often requiring clinicians to adopt awkward body positioning, reposition equipment/people to gain access to the equipment, or place supplies in a precarious position due to lack of appropriate work surface.	Latent conditions Person works around immediate obstruction to access other equipment/supplies Person assumes awkward body positioning due to obstructed access to equipment/supplies Equipment/supplies placed in precarious position due to lack of access to an appropriate work surface Other
Obstructed access to equipment or supplies^a,c,f,g		Active failures Delay in care due to obstructed access to equipment/supplies Care proceeds without desired equipment/supplies Supplies fall on floor after being placed in a precarious position Other
Obstructed access to patient^a,c,f,g	Obstructions immediately at or around the bedside impede direct access to the patient as required to provide care. Clinicians may have to reposition equipment or team members, move around the obstruction, or assume awkward body positioning to overcome this type of hazard.	Latent conditions Equipment repositioned to get to patient Person assumes awkward body positioning due to obstructed access to patient Person moves around immediately adjacent obstructed to get to patient Reposition someone else who is immediately adjacent to the patient in order to access the patient Other
Obstructed access to patient^a,c,f,g		Active failures Delay in care due to obstructed access to patient Inadequate care delivery due to lack of access to patient Other
Obstructed pathway^a,c,f,g	Clinicians or equipment are unable to travel directly between two points as required to execute their tasks; they may have to take a circuitous path due to barriers or obstructions including fixed room elements, equipment, or other people or may have to move barriers out of their way.	Latent conditions Significant change in path around person or equipment Person moves out of the path Equipment is moved out of the path Other
Obstructed pathway^a,c,f,g		Active failures Delays in care due to obstructed path Harm/damage to person or equipment (e.g., harm from collision during translational movement) Other
Obstructed visibility^d,f	Clinicians are unable to see clocks or monitors as required for their role and may have to move or interrupt their task to gain line of sight.	Latent conditions Person moves or repositions in order to gain line of sight, leading to interruption of/distraction from another task Person moves significantly to gain line of sight Person moves equipment to gain line of sight Person delegates someone as proxy to view monitor/equipment in question for them Other
Obstructed visibility^d,f		Active failures Work without line of sight Miss critical changes on monitor Other
Infection risk^e	Due to the setup of the physical environment, including fixed and variable elements, sterile technique is compromised or sterile equipment/field is at risk of contamination.	Latent conditions Nonsterile person reaches across sterile field without touching sterile field or equipment Sterile items placed in position that compromised sterility Nonsterile person walks between two sterile surfaces (such as OR table and sterile person, or OR table and draped bed) Other
Infection risk^e		Active failures Sterile field is contaminated Sterile equipment is contaminated Sterile person is contaminated (such as their hands, core, etc.) Other

Notes. Table 2 provides a brief overview of each hazard category and associated operational definitions, including descriptions of both latent safety threats and active failures. Links between each hazard categories and relevant Agency for Healthcare Research and Quality/Center for Health Design safe design principles are noted. Categories include ^a minimize environmental hazards, ^b improve visibility, ^c minimizing staff fatigue, ^d communication breakdown, ^e control/eliminate sources of infection, ^f provide safe delivery of care, and ^g provide efficient delivery of care, OR: operating room.

Assessment of IRR: Paired Raters

Interrater agreement between two pairs of raters (each consisting of one clinical and one nonclinical coder) was assessed across 48 video segments. Both nonclinical raters (K.G., L.D.) and one clinician (C.A.) were involved in tool development, while the other clinician (G.U., PICU nurse) was a novice rater who underwent training utilizing a frame of reference training strategy (Keown-Gerrard & Sulsky, 2001; Roch et al., 2012). One experienced rater reviewed operational definitions and sample video clips representative of each hazard category with the novice coder. The novice coder and experienced clinical coder then scored videos using the established coding protocol followed by detailed review of all instances of rater nonagreement to allow feedback on rating practices from the experienced to the novice rater. A third video was reviewed by the clinical rater pair for formal assessment of IRR among individual clinical raters.

Overall reliability was calculated by combining all six hazard categories for a total of 315 subjects, while reliability for each individual hazard category was calculated across a minimum of 48 segments (range: 48–68 per category). Overall, the percentage of segments with identified hazards ranged from 8% to 13% across all raters. Due to extremely low occurrence of observable hazards in the infection risk category, an additional 18 segments were coded (Table 2).

Landis and Koch’s (Landis & Koch, 1977) benchmark ranges are used to interpret the relative strength of agreement: <0.00 = poor; 0.00–0.20 = slight; 0.21–0.40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = substantial; and 0.80–1.00 = almost perfect (Landis & Koch, 1977). Overall IRR using Gwet’s AC₁ was 0.89 (95% confidence interval [0.843, 0.929]), indicating almost perfect agreement. IRR across the individual hazard categories ranged from substantial agreement: obstructed path = 0.76 [0.59, 0.93] to almost perfect agreement: obstructed access to patient = 0.93 [0.84, 1.0]; infection risk = 0.92 [0.84, 0.99]; poor visibility = 0.91 [0.81, 1.0]; slip, trip, fall, or injury risks = 0.89 [0.779, 1.0]; and obstructed access to equipment = 0.88 [0.764, 1.0]. Comprehensive results, including percent agreement and Cohen’s κ, are summarized in Table 4a.

Assessment of IRR: Individual Clinical Raters

IRR was also assessed between two individual clinician raters. Overall, reliability was calculated across 294 subjects, resulting in almost perfect agreement (AC₁ = 0.85; 95% CI [0.8, 0.903]). IRR for individual hazard categories ranged from substantial agreement (two of the six hazard categories): slip, trip, fall risks = 0.68 [0.47, 0.88]; obstructed access to equipment = 0.77 [0.6, 0.94]; to almost perfect agreement (four of the six hazard categories): poor visibility = 0.93 [0.85, 1.0]; obstructed access to patient = 0.93 [0.85, 1.0]; infection risk = 0.88 [0.78, 0.99]; and obstructed path = 0.87 [0.75, 0.99]. Comprehensive results are summarized in Table 4b.

Table 4.

Assessment of Interrater Reliability: (a) Paired Raters and (b) Individual Clinical Raters.

Hazard Category	Gwet’s AC₁	Gwet’s AC₁ 95% Confidence Interval	Percent Agreement	Percent Agreement 95% Confidence Interval	Cohen’s κ	Cohen’s κ 95% Confidence Interval
(a) Paired raters
Overall	0.88633	[0.843, 0.929]	0.90794	[0.876, 0.94]	0.51720	[0.364, 0.671]
Slip, trip, fall, or injury risks	0.89189	[0.779, 1]	0.91837	[0.839, 0.998]	0.66780	[0.357, 0.979]
Obstructed access to equipment	0.88462	[0.764, 1]	0.91667	[0.836, 0.998]	0.70370	[0.428, 0.979]
Obstructed access to patient	0.92941	[0.844, 1]	0.93878	[0.869, 1]	0.53918	[0.062, 1]
Obstructed path	0.76177	[0.593, 0.93]	0.82692	[0.721, 0.933]	0.40000	[0.105, 0.695]
Poor visibility	0.90776	[0.811, 1]	0.91837	[0.839, 0.998]	0.29496	[−0.22, 0.81]
Infection prevention risk	0.91851	[0.843, 0.994]	0.92647	[0.863, 0.99]	0.25439	[−0.198, 0.707]
(b) Individual clinical raters
Overall	0.85181	[0.8, 0.903]	0.88095	[0.844, 0.918]	0.39761	[0.236, 0.559]
Slip, trip, fall, or injury risks	0.67548	[0.473, 0.878]	0.79245	[0.68, 0.905]	0.43122	[0.147, 0.716]
Obstructed access to equipment	0.76923	[0.596, 0.943]	0.83333	[0.724, 0.943]	0.40741	[0.065, 0.749]
Obstructed access to patient	0.93065	[0.847, 1]	0.93750	[0.866, 1]	0.37931	[−0.174, 0.932]
Obstructed path	0.87256	[0.752, 0.993]	0.89796	[0.81, 0.986]	0.48852	[0.089, 0.888]
Poor visibility	0.93347	[0.853, 1]	0.93750	[0.866, 1]	−0.02857	[−0.07, 0.013]
Infection risk	0.88442	[0.775, 0.994]	0.89583	[0.806, 0.985]	−0.05263	[−0.105, −0.001]

Note. Table 4 presents the results of interrater reliability assessment for the tool as a whole and by individual hazard category using Gwet’s AC1, percent agreement, and Cohen’s κ.

Discussion

Design of safe healthcare facilities requires deliberate attention to latent conditions at all levels of design, from overall space planning to consideration of key adjacencies and to design of individual care spaces. We report here on the development and establishment of IRR for the HART, designed to aid identification of hazards related to physical workspaces during preconstruction simulation-based design testing. Applied to audiovisual recordings of in situ simulations in the ICU setting, HART allows trained raters to identify specific instances of hazards across six distinct categories with a high degree of IRR, facilitating thorough and objective identification of environmental hazards that may adversely impact patients and clinicians. This instrument also augments the available simulation-based facility design evaluation tool kit by introducing a means of quantitative assessment of the effectiveness of design modifications through application to simulations conducted before and after design modifications.

Applied to audiovisual recordings of in situ simulations in the ICU setting, HART allows trained raters to identify specific instances of hazards across six distinct categories with a high degree of IRR, facilitating thorough and objective identification of environmental hazards that may adversely impact patients and clinicians. This instrument also augments the available simulation-based facility design evaluation tool kit by introducing a means of quantitative assessment of the effectiveness of design modifications through application to simulations conducted before and after design modifications.

HART is intended to identify the ways in which specific elements of room design may constrain or threaten provision of care. For example, a smaller room may induce crowding of equipment and providers in key locations, thereby restricting access to the bedside or hindering providers in completion of clinical tasks that require them to move about the room. Similarly, suboptimal positioning of electrical outlets may cause cords to span crucial clinical workspace, thereby increasing risk of provider injury (Brogmus et al., 2007; Joseph et al., 2018; Shah et al., 2021; Wiegmann et al., 2010). Its basis in functional analysis supports efforts to optimize delivery of care by providing a mechanism for objective, quantitative evaluation with successive design changes.

Retaining Human Factors Framework for Coding by Clinicians and Simulation Experts

Human factors engineering, in which detailed observation of human–human, human–technology, and human–system interactions is used to identify factors that impede work or compromise safety, has been effectively applied to identify risk in clinical environments (Carayon, 2012; Carayon et al., 2014; Holden et al., 2013). In high-risk healthcare settings, teams with both clinical expertise and specific training in evaluation of workspace design (common to human factors engineers; Wickens et al., 2014) have optimized the output of such work (Gurses et al., 2012). Recognizing the important contribution of both clinical and human factors perspectives, HART was developed by a multidisciplinary team with both clinical and nonclinical perspectives. Through standardization and optimization of coding practices and refinement of operational definitions, we were able to achieve excellent IRR between these multidisciplinary rating pairs. Recognizing that many institutions do not have embedded human factors experts (Hignett et al., 2018; Wears, 2015), we also established IRR between individual clinician raters. The rigorous and structured nature of the tool thereby indirectly lends a human factors lens when utilized by individual trained clinician raters.

Video Coding Protocol and Detailed Framework Improve Hazard Identification

HART enables functional analysis of interactions between providers and elements of their environment (e.g., equipment, technology, layout, and overall space) yielding quantitative data on the presence of hazards. The use of video review eliminates recall bias inherent in identification of hazards through postevent debriefing, and a standardized coding protocol improves event capture and counters salience bias (Gurses et al., 2012; Haidet et al., 2009). Clinicians accustomed to adaptation may not always recognize suboptimal design if they are able to develop workarounds; however, these same adaptations may contribute to provider fatigue and increased risk to both patients and providers (Brogmus et al., 2007; Joseph et al., 2018; Shah et al., 2021; Wiegmann et al., 2010). Facility design requires a more critical lens, allowing design teams to construct clinical environments that obviate or significantly reduce the need for workarounds. Finally, formal operational definitions allow trained clinician raters to apply the tool, improving usability by centers without dedicated human factors experts while retaining a systematic functional analysis perspective.

Alignment of HART With Existing Safety Models

Previous work in simulation-based design testing has leveraged the evidence-based Safe Design Principles as a framework for identifying and categorizing latent safety threats (Colman, Edmond, et al., 2020; Joseph et al., 2012). This framework outlines broad guiding principles for safe design, such as avoiding staff fatigue, minimizing environmental hazards, and promoting standardization. However, the Safe Design Principles do not provide specific, concrete guidance on how to identify design features that either support or negate these principles. By supplying operational definitions to anchor observation in six domains, each of which links to one more of the Safe Design Principles, HART facilitates identification of specific elements in the environment, including interactions between people and equipment, the environment, and technology, that can be reworked to better align with the Safe Design Principles. For example, repeated observation of clinicians’ circuitous routes and awkward positioning to reach and manage a patient’s airway during a critical event provides insight into how the environment might be optimized to reduce staff fatigue. Likewise, observed interactions with cords and tubing can identify opportunities to reduce slip, trip, or fall risk.

The hazard categories described in HART are divergent from others in the current literature. HART categories are explicitly designed to support observation of hazards through a functional lens examining the relationship of a person to the physical environment of care during an episode of work. Previous studies focus on categorizing hazards according to existing system-based safety frameworks, such as Safe Design Principles and SEIPS Model 2.0 (Carayon et al., 2006; Colman, Dalpiaz, et al., 2020). While these approaches might appear dissimilar, qualitative data collected as part of the HART coding process (see below) can readily be mapped onto existing safety frameworks, yielding complementary approaches.

Future Directions: Application of HART to Support Hazard Reduction

In addition to providing a concrete link between Safe Design Principles and the manifestation of safety threats in the clinical environment, HART also increases the rigor that may be applied to quantitative evaluation of hazards. While this article focuses on the development of the tool and establishment of IRR, application of the tool yields a score in hazards per minute, calculated by dividing the total number of hazards by the total duration of the simulation. Others have leveraged real-time observation and video review (Mackenzie et al., 2007; Patterson et al., 2013) of simulation to identify hazards, which were then scored using Failure Modes Effects Analysis (FMEA; Colman et al., 2019b; Davis et al., 2008; DeRosier et al., 2002; Geis et al., 2011; Thornton et al., 2011). This methodology yields a quantitative assessment that can be used to prioritize highest impact changes and track hazard mitigation across repeated simulation-based assessments of design changes. However, identification of failure modes and assignment of FMEA scores remain largely subjective and can be challenging for diverse stakeholder groups to apply (Habraken et al., 2009; Liu et al., 2020). HART provides both objective identification and measured rather than estimated frequency of hazards, potentially improving accuracy of hazard assessment, prioritization of targets for intervention, and evaluation of mitigation strategies. Both identification of failure modes and difficulty operationalizing risk assessment have been cited as challenges in application of FMEA (Habraken et al., 2009).

While the current article describes the development and establishment of IRR of HART, ongoing work focuses on determining if application of HART can accurately distinguish between rooms with different design features, and whether these differences correlate with important clinical performance metrics. In addition, development of the methodology to incorporate HART into existing frameworks for evaluation of SBDT, such as FMEA, to guide change is ongoing.

Limitations

HART was developed and IRR established utilizing simulations from PICU environments. Generalizability of the tool to other clinical environments has not yet been formally assessed. Applicability of the tool to acute patient care environments with highly specific equipment, technology, and team requirements (such as operating rooms and emergency departments) will be of particular importance. Additionally, the tool has not been evaluated for its applicability to routine patient care encounters (as opposed to high-risk encounters), including in the outpatient arena, though healthcare facility design should be evaluated for both high-risk and routine uses (Joseph et al., 2012). Establishment of IRR used video recordings of PICU-ISS recorded from two camera angles; although multiple pilot simulations were conducted to optimize audio–video capture, there remains the possibility for “blind spots” or incomplete audio that could impact coding fidelity. While the use of additional cameras might allow some incremental improvement in event capture, this would come at a cost of a significant increase in coding time. Recognizing this trade-off, we made a deliberate decision to limit to two optimized camera angles. For others who might wish to utilize the HART tool, attention to optimal camera angles will be a key part of simulation setup.

The objective of HART is to capture a broad range of potential hazards related to the clinical workspace through the application of operational definitions that provide specific guidance on hazard identification. While the tool was designed for breadth, it is not possible, nor is it the intention of the operational definitions, to describe every possible hazard that could be encountered. Therefore, it is possible that hazards will be missed due to the application of the structured operational definitions. However, development of a more structured coding process and creation of more specific operational definitions in practice reduced the instances of event nondetection during the development phase of this tool.

It is important to note that HART was purposefully created to identify hazards related to design of individual clinical spaces such as ICU rooms. HART supports implementation of a number of evidence based safe design principles by guiding specific hazard detection through concrete operational definitions. However, complete evaluation of facility design also requires exploration of broader system elements related to overall space programming, adjacencies, and how these larger design elements support care processes beyond the level of individual rooms. Designed HART does not support this broader healthcare facility evaluation. Others have noted the importance of using multiple modalities in the evaluation of healthcare facility design (Joseph et al., 2012), and HART is one element of this broader tool kit.

Summary

HART is a structured observational tool that can be applied to identify specific hazards in the built environment that introduce risk to patients and providers during episodes of simulated emergency care. The tool can be applied with a high degree of IRR by either paired multidisciplinary raters or individual clinical raters through review of audiovisual recordings of in situ simulations.

Implications for Practice

The HART allows structured review of audiovisual recordings of simulations to evaluate facility design by trained raters, potentially mitigating gaps in hazard identification through participant debriefing and decreasing reliance on human factors experts.

Attention to audiovisual recording setup and a structured rating approach are key to reliable application of the tool.

The tool performs well when utilized either paired clinical and nonclinical raters or single clinical raters.

Hazards identified through application of HART could be used to guide changes in facility design to minimize hazards in the built environment.

Supplemental Material

Supplemental Material, sj-pdf-1-her-10.1177_19375867231188151 - Hazard Assessment and Remediation Tool for Simulation-Based Healthcare Facility Design Testing

Supplemental Material, sj-pdf-1-her-10.1177_19375867231188151 for Hazard Assessment and Remediation Tool for Simulation-Based Healthcare Facility Design Testing by Marlena Smith-Millman, Lorraine Daniels, Katie Gallagher, Sarah Aspinwall, Howard Brightman, Gina Ubertini, Gaia Uman Borrero, Lobsang Palmo, Peter Weinstock and Catherine Allan in HERD: Health Environments Research & Design Journal

Supplemental Material

Supplemental Material, sj-pdf-2-her-10.1177_19375867231188151 - Hazard Assessment and Remediation Tool for Simulation-Based Healthcare Facility Design Testing

Supplemental Material, sj-pdf-2-her-10.1177_19375867231188151 for Hazard Assessment and Remediation Tool for Simulation-Based Healthcare Facility Design Testing by Marlena Smith-Millman, Lorraine Daniels, Katie Gallagher, Sarah Aspinwall, Howard Brightman, Gina Ubertini, Gaia Uman Borrero, Lobsang Palmo, Peter Weinstock and Catherine Allan in HERD: Health Environments Research & Design Journal

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: This work was supported by the Agency for Healthcare Research and Quality (grant number R18HS025927).

ORCID iDs

Marlena Smith-Millman,MPH

Lorraine Daniels,PhD

Katie Gallagher,MA

Gaia Uman Borrero,BA

Catherine Allan,MD

Supplemental Material

The supplemental material for this article is available online.

References

Adler

M. D.

Mobley

B. L.

Eppich

W. J.

Lappe

Green

Mangold

(2018). Use of simulation to test systems and prepare staff for a new hospital transition. Journal of Patient Safety, 14(3), 143–147. https://doi.org/10.1097/PTS.0000000000000184

Bayramzadeh

Joseph

Allison

Shultz

Abernathy

Group

R. O. S

. (2018). Using an integrative mock-up simulation approach for evidence-based evaluation of operating room design prototypes. Applied Ergonomic, 70, 288–299. https://doi.org/10.1016/j.apergo.2018.03.011

Birnbach

D. J.

Nevo

Scheinman

S. R.

Fitzpatrick

Shekhter

Lombard

J. L.

(2010). Patient safety begins with proper planning: a quantitative method to improve hospital design. Quality & Safety in Health Care, 19(5), 462–465. https://doi.org/10.1136/qshc.2008.031013

Blike

G. T.

Christoffersen

Cravero

J. P.

Andeweg

S. K.

Jensen

(2005). A method for measuring system safety and latent errors associated with pediatric procedural sedation. Anesthesia & Analgesia, 101(1), 48–58, Table of contents. https://doi.org/10.1213/01.ANE.0000152614.57997.6C

Braun

V. C. V

. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.

Broberg

Andersen

Seim

(2011). Participatory ergonomics in design processes: The role of boundary objects. Applied Ergonomics, 42(3), 464–472. https://doi.org/10.1016/j.apergo.2010.09.006

Brogmus

Leone

Butler

Hernandez

(2007). Best practices in OR suite layout and equipment choices to reduce slips, trips, and falls. AORN Journal, 86(3), 384–394; quiz 395–388. https://doi.org/10.1016/j.aorn.2007.06.003

Carayon

. (2012). Handbook of human factors and ergonomics in health care and patient safety. CRC Press. https://doi.org/10.1002/9781119636113.fmatter

Carayon

Schoofs Hundt

Karsh

B. T.

Gurses

A. P.

Alvarado

C. J.

Smith

Flatley Brennan

(2006). Work system design for patient safety: The SEIPS model. Quality & Safety in Health Care, 15(Suppl 1), i50–58. https://doi.org/10.1136/qshc.2005.015842

10.

Carayon

Xie

Kianfar

(2014). Human factors and ergonomics as a patient safety practice. BMJ Quality & Safety, 23(3), 196–205. https://doi.org/10.1136/bmjqs-2013-001812

11.

Colman

Dalpiaz

Walter

Chambers

M. S.

Hebbar

K. B.

(2020). SAFEE: A debriefing tool to identify latent conditions in simulation-based hospital design testing. Advances in Simulation (Lond), 5, 14. https://doi.org/10.1186/s41077-020-00132-2

12.

Colman

Doughty

Arnold

Stone

Reid

Dalpiaz

Hebbar

K. B.

(2019a). Simulation-based clinical systems testing for healthcare spaces: From intake through implementation. Advances in Simulation, 4(1), 19. https://doi.org/10.1186/s41077-019-0108-7

13.

Colman

Edmond

M. B.

Dalpiaz

Walter

Miller

D. C.

Hebbar

(2020). Designing for patient safety and efficiency: Simulation-based hospital design testing. Health Environments Research & Design Journal, 13(4), 68–80. https://doi.org/10.1177/1937586720921777

14.

Colman

Stone

Arnold

Doughty

Reid

Younker

Hebbar

K. B.

(2019b). Prevent safety threats in new construction through integration of simulation and FMEA. Pediatric Quality & Safety, 4(4), e189. https://doi.org/10.1097/pq9.0000000000000189

15.

Davis

Riley

Gurses

A. P.

Miller

Hansen

(2008). Failure modes and effects analysis based on in situ simulations: A methodology to improve understanding of risks and failures. In Henriksen

Battles

J. B.

Keyes

M. A.

Grady

M. L.

(Eds.), Advances in patient safety: New directions and alternative approaches (Vol. 3: Performance and tools). https://www.ncbi.nlm.nih.gov/pubmed/21249922

16.

DeRosier

Stalhandske

Bagian

J. P.

Nudell

(2002 ). Using health care failure mode and effect analysis: The VA national center for patient safety’s prospective risk analysis system. Joint Commission Journal on Quality Improvement, 28(5), 248–267, 209. https://doi.org/10.1016/s1070-3241(02)28025-6

17.

Deutsch

(2017). Bridging the gap between work-as-imagined and work-as-done. https://psnet.ahrq.gov/issue/bridging-gap-between-work-imagined-and-work-done

18.

Fan

Petrosoniak

Pinkney

Hicks

White

Almeida

A. P.

Campbell

McGowan

Gray

Trbovich

(2016). Study protocol for a framework analysis using video review to identify latent safety threats: trauma resuscitation using in situ simulation team training (TRUST). BMJ Open, 6(11), e013683. https://doi.org/10.1136/bmjopen-2016-013683

19.

Feinstein

A. R.

Cicchetti

D. V.

(1990). High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-l

20.

Fraser

K. L.

Ayres

Sweller

(2015). Cognitive load theory for the design of medical simulations. Simulation in Healthcare, 10(5), 295–307. https://doi.org/10.1097/SIH.0000000000000097

21.

Garmer

Ylvén

MariAnne Karlsson

I. C.

(2004). User participation in requirements elicitation comparing focus group interviews and usability tests for eliciting usability requirements for medical equipment: a case study. International Journal of Industrial Ergonomics, 33(2), 85–98. https://doi.org/10.1016/j.ergon.2003.07.005

22.

Geis

G. L.

Pio

Pendergrass

T. L.

Moyer

M. R.

Patterson

M. D.

(2011). Simulation to assess the safety of new healthcare teams and new facilities. Simulation in Healthcare, 6(3), 125–133. https://doi.org/10.1097/SIH.0b013e31820dff30

23.

Gignon

Amsallem

Ammirati

(2017). Moving a hospital: simulation—A way to co-produce safety healthcare facilities. International Journal of Occupational Safety and Ergonomics, 23(4), 589–591. https://doi.org/10.1080/10803548.2016.1270543

24.

Gurses

A. P.

Kim

Martinez

E. A.

Marsteller

Bauer

Lubomski

L. H.

Pronovost

P. J.

Thompson

(2012). Identifying and categorising patient safety hazards in cardiovascular operating rooms using an interdisciplinary approach: a multisite study. BMJ Quality & Safety, 21(10), 810–818. https://doi.org/10.1136/bmjqs-2011-000625

25.

Gwet

(2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC.

26.

Habraken

M. M.

Van der Schaaf

T. W.

Leistikow

I. P.

Reijnders-Thijssen

P. M.

(2009). Prospective risk analysis of health care processes: A systematic evaluation of the use of HFMEA in Dutch health care. Ergonomics, 52(7), 809–819. https://doi.org/10.1080/00140130802578563

27.

Haidet

K. K.

Tate

Divirgilio-Thomas

Kolanowski

Happ

M. B.

(2009). Methods to improve reliability of video-recorded behavioral data. Research in Nursing & Health, 32(4), 465–474. https://doi.org/10.1002/nur.20334

28.

Hignett

Albolino

Catchpole

(2018). Health and social care ergonomics: Patient safety in practice. Ergonomics, 61(1), 1–4. https://doi.org/10.1080/00140139.2017.1386454

29.

Holden

R. J.

Carayon

Gurses

A. P.

Hoonakker

Hundt

A. S.

Ozok

A. A.

Rivera-Rodriguez

A. J.

(2013). SEIPS 2.0: A human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 56(11), 1669–1686. https://doi.org/10.1080/00140139.2013.838643

30.

Joseph

Bayramzadeh

Zamani

Rostenberg

(2018). Safety, performance, and satisfaction outcomes in the operating room: A literature review. Health Environments Research & Design Journal, 11(2), 137–150. https://doi.org/10.1177/1937586717705107

31.

Joseph

Quan

Taylor

Jelen

(2012). Designing for patient safety: Developing methods to integrate patient safety concerns in the design process. Center for Healthcare Design, 105–116.

32.

Kaba

Barnes

(2019). Commissioning simulations to test new healthcare facilities: a proactive and innovative approach to healthcare system safety. Advances in Simulation(Lond), 4, 17. https://doi.org/10.1186/s41077-019-0107-8

33.

Keown-Gerrard

J. L.

Sulsky

L. M.

(2001). The effects of task information training and frame-of-reference training with situational constraints on rating accuracy. Human Performance, 14(4), 305–320. https://doi.org/10.1207/S15327043HUP1404_2

34.

Kottner

Audige

Brorson

Donner

Gajewski

B. J.

Hrobjartsson

Roberts

Shoukri

Streiner

D. L.

(2011). Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. International Journal of Nursing Studies, 48(6), 661–671. https://doi.org/10.1016/j.ijnurstu.2011.01.016

35.

Kroemer

K. H.

(1989). Cumulative trauma disorders: Their recognition and ergonomics measures to avoid them. Applied Ergonomics, 20(4), 274–280. https://doi.org/10.1016/0003-6870(89)90190-7

36.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://www.ncbi.nlm.nih.gov/pubmed/843571

37.

Liu

Zhu

Huang

Cheng

(2020). Using healthcare failure mode and effect analysis as a method of vaginal birth after caesarean section management. Journal of Clinical Nursing, 29(1-2), 130–138. https://doi.org/10.1111/jocn.15069

38.

Mackenzie

C. F.

Xiao

F. M.

Seagull

F. J.

Fitzgerald

(2007). Video as a tool for improving tracheal intubation tasks for emergency medical and trauma care. Annals of Emergency Medicine, 50(4), 436–442, 442.e431. https://doi.org/10.1016/j.annemergmed.2007.06.487

39.

Mangold. (2020). INTERACT User Guide. www.mangold-international.com

40.

Nickson

C. P.

Petrosoniak

Barwick

Brazil

(2021). Translational simulation: from description to action. Advances in Simulation (Lond), 6(1), 6. https://doi.org/10.1186/s41077-021-00160-6

41.

Oakley

Stocker

Staubli

Young

(2006). Using video recording to identify management errors in pediatric trauma resuscitation. Pediatrics, 117(3), 658–664. https://doi.org/10.1542/peds.2004-1803

42.

Patterson

M. D.

Geis

G. L.

Falcone

R. A.

LeMaster

Wears

R. L.

(2013). In situ simulation: detection of safety threats and teamwork training in a high risk emergency department. BMJ Quality & Safety, 22(6), 468–477. https://doi.org/10.1136/bmjqs-2012-000942

43.

Petrosoniak

Fan

Hicks

C. M.

White

McGowan

Campbell

Trbovich

(2021). Trauma resuscitation using in situ simulation team training (TRUST) study: Latent safety threat evaluation using framework analysis and video review. BMJ Quality & Safety, 30(9), 739–746. https://doi.org/10.1136/bmjqs-2020-011363

44.

Reiling

(2006). Safe design of healthcare facilities. Quality & Safety in Health Care, 15 Suppl 1(Suppl 1), i34–40. https://doi.org/10.1136/qshc.2006.019422

45.

Roch

S. G.

Woehr

D. J.

Mishra

Kieszczynska

(2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85(2), 370–395. https://doi.org/10.1111/j.2044-8325.2011.02045.x

46.

Rogers

Buckheit

Ostendorf

(2013). Ergonomics and nursing in hospital environments. Workplace Health & Safety, 61(10), 429–439. https://doi.org/10.1177/216507991306101003

47.

Shah

McGowan

Petrosoniak

(2021). Latent safety threat identification during in situ simulation debriefing: A qualitative analysis. BMJ Simulation & Technology Enhanced Learning, 7(4), 194–198. https://doi.org/10.1136/bmjstel-2020-000650

48.

Taylor

Hignett

Joseph

(2014). The environment of safe care: considering building design as one facet of safety. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 3(1), 123–127. https://doi.org/10.1177/2327857914031020

49.

Taylor

Quan

Joseph

(2015). Testing a tool to support safety in healthcare facility design. Procedia Manufacturing, 3, 136–143. https://doi.org/10.1016/j.promfg.2015.07.118

50.

Thornton

Brook

O. R.

Mendiratta-Lala

Hallett

D. T.

Kruskal

J. B.

(2011). Application of failure mode and effect analysis in a radiology department. Radiographics, 31(1), 281–293. https://doi.org/10.1148/rg.311105018

51.

Wears

(2015). The rise and fall of patient safety: Implications for human factors. Proceedings of the HFES 2015 International Symposium on Human Factors Ergon Healthcare, 26–29.

52.

Wickens

C. D.

Gordon

S. E.

Liu

Lee

(2014). An introduction to human factors engineering. Pearson Prentice Hall.

53.

Wiegmann

D. A.

Eggman

A. A.

Elbardissi

A. W.

Parker

S. H.

Sundt

T. M.

, 3 rd. (2010). Improving cardiac surgical care: A work systems approach. Applied Ergonomics, 41(5), 701–712. https://doi.org/10.1016/j.apergo.2009.12.008

54.

Wingler

Machry

Bayramzadeh

Joseph

Allison

(2019). Comparing the effectiveness of four different design media in communicating desired performance outcomes with clinical end users. Health Environments Research & Design Journal, 12(2), 87–99. https://doi.org/10.1177/1937586718796626

55.

Wongpakaran

Wedding

Gwet

K. L.

(2013). A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology, 13(1), 61. https://doi.org/10.1186/1471-2288-13-61

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.57 MB

0.23 MB

0.00 MB