Test the automation transparency design principle using a full-scope nuclear power plant simulator.
Background
Automation transparency is a long-held human factors design principle espousing that the responsibilities, capabilities, goals, activities, and/or effects of automation should be directly observable in the human–system interface. The anticipated benefits of transparency include more effective reliance, more appropriate trust, better understanding, and greater user satisfaction. Transparency has enjoyed a recent upsurge in use in the context of human interaction with agent-oriented automation.
Method
Three full-scope nuclear power plant simulator studies were conducted with licensed operating crews. In the first two experiments, transparency was implemented for interlocks, controllers, limitations, protections, and automatic programs that operate at the local component level of the plant. In the third experiment, procedure automation assumed control of plant operations and was represented in dedicated agent displays.
Results
Results from Experiments 1 and 2 appear to validate the human performance benefits of automation transparency for automation at the component level. However, Experiment 3 failed to replicate these findings for automation that assumed control for executing procedural actions.
Conclusion
Automation transparency appears to yield expected benefits for component-level automation, but caution is warranted in generalizing the design principle to agent-oriented automation.
Application
The automation transparency design principle may offer a powerful means of compensating for the detrimental impacts of hidden automation influence at the component level of complex systems. However, system developers should exercise caution in assuming that the principle extends to agent-oriented automation.
ChristoffersenK.WoodsD. (2002). How to make automated systems team players. Advances in Human Performance and Cognitive Engineering Research, Volume, 2, 1–12. doi:10.1016/s1479-3601(02)02003-9
9.
Dell Inc. (2015). Dell statistica (data analysis software system) (Version 13). Available from dell.com
10.
EndsleyM. R. (2000). Direct measurement of situation awareness: Validity and use of SAGAT. In EndsleyM. R.GarlandD. J. (Eds.), Situation awareness analysis and measurement (pp. 147–173). Mahwah, NJ: Lawrence Erlbaum.
11.
FernE. F.MonroeK. B. (1996). Effect-size estimates: Issues and problems in interpretation. Journal of Consumer Research, 23, 89–105.
12.
FernandesA.BraarudP. Ø. (2015). Exploring measures of workload, situation awareness, and task performance in the Main Control Room. Procedia Manufacturing, 3, 1281–1288.
13.
HeitzR. P. (2014). The speed-accuracy tradeoff: History, physiology, methodology, and behavior. Frontiers in Neuroscience, 8, 150.
14.
HoffK. A.BashirM. (2015). Trust in automation. Human Factors, 57, 407–434. doi:10.1177/0018720814547570
15.
IEEE 1786-2011. (2011). IEEE human factors guide for applications of computerized operating procedure systems at nuclear power generating stations and other nuclear facilities. New York, NY: Institute of Electrical and Electronic Engineers.
16.
JamiesonG. A.SkraaningG. (2019). The absence of degree of automation trade-offs in complex work settings. Human Factors. Advance online publication. doi:10.1177/0018720819842709
17.
JohnsonM.BradshawJ. M.FeltovichP. J.JonkerC. M.Van RiemsdijkM. B.SierhuisM. (2014). Coactive design: Designing support for interdependence in joint activity. Journal of Human-Robot Interaction, 3(1), 43. doi:10.5898/JHRI.3.1.Johnson
18.
KirkR. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks Cole.
19.
LauN.JamiesonG. A.SkraaningG. (2016a). Empirical evaluation of the Process Overview Measure for assessing situation awareness in process plants. Ergonomics, 59, 393–408. doi:10.1080/00140139.2015.1080310
20.
LauN.JamiesonG. A.SkraaningG. (2016b). Situation awareness acquired from monitoring process plants—The process overview concept and measure. Ergonomics, 59, 976–988. doi:10.1080/00140139.2015.1100329
21.
LauN.SkraaningG.EitrheimM. H.KarlssonT.NihlwingC.JamiesonG. (2011). Situation awareness in monitoring nuclear power plants—The process overview concept and measure (HWR-954). Halden, Norway: OECD Halden Reactor Project.
22.
LauN.SkraaningG.KarlssonT.NihlwingC.JamiesonG. (2011). The process overview measure: Methodological developments to enhance inter-rater reliability (HWR-971). Halden, Norway: OECD Halden Reactor Project.
23.
LyonsJ. B. (2013). Being transparent about transparency: A model for human-robot interaction. In Trust and Autonomous Systems: Papers from the 2013 AAAI Spring Symposium, pp. 48–53. doi:10.1021/ja00880a025
24.
LyonsJ. B.KoltaiK. S.HoN. T.JohnsonW. B.SmithD. E.ShivelyR. J. (2016). Engineering trust in complex automated systems. Ergonomics in Design, 24(1), 13–17. doi:10.1177/1064804615611272
25.
LyonsJ. B.SadlerG. G.KoltaiK.BattisteH.HoN. T.HoffmannL. C.. . . ShivelyR. (2017). Shaping trust through transparent design: Theoretical and experimental guidelines. In Savage-KnepshieldP.ChenJ. (Eds.), Advances in intelligent systems and computing (pp. 127–136). Cham, Switzerland: Springer. doi:10.1007/978-3-319-41959-6_11
26.
MercadoJ. E.RuppM. A.ChenJ. Y.BarnesM. J.BarberD.ProcciK. (2016). Intelligent agent transparency in human–agent teaming for multi-UxV management. Human Factors, 58, 401–415. doi:10.1177/0018720815621206
27.
NormanD. A. (1990). The “problem” with automation: Inappropriate feedback and interaction, not “over-automation.”Philosophical Transactions of the Royal Society of London B: Biological Sciences, 327, 585–593.
28.
SarterN. B.WoodsD. D. (1997). Team play with a powerful and independent agent: Operational experiences and automation surprises on the Airbus A-320. Human Factors, 39, 553–569. doi:10.1518/001872097778667997
29.
SelkowitzA. R.LakhmaniS. G.LariosC. N.ChenJ. Y. (2016, September). Agent transparency and the autonomous squad member. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 60, No. 1, pp. 1319–1323). Los Angeles, CA: SAGE.
30.
SeongY.BisantzA. M. (2008). The impact of cognitive feedback on judgment performance and trust with decision aids. International Journal of Industrial Ergonomics, 38, 608–625.
31.
SkjerveA. B. M.AndresenG.SaarniR.SkraaningG. (2001). How automation malfunctions influence operator performance. Study plan for the HCA-2000 experiment (HWR-659). Halden, Norway: OECD Halden Reactor Project.
32.
SkjerveA. B. M.ByeA. (2011). Simulator-based human factors studies across 25 years. London, England: Springer.
33.
SkjerveA. B. M.SkraaningG. (2004). The quality of human-automation cooperation in human-system interface for nuclear power plants. International Journal of Human-computer Studies, 61, 649–677. doi:10.1016/j.ijhcs.2004.06.001
34.
SkjerveA. B. M.StrandS.SaarniR.SkraaningG. (2002). The influence of automation malfunctions and interface design on operator performance. The HCA-2001 experiment (HWR-686). Halden, Norway: OECD Halden Reactor Project.
35.
SkjerveA. B. M.StrandS.SkraaningG.NihlwingC.HelgarS.OlsenA.SvengrenH. (2005). The extended teamwork 2004/2005 exploratory study. Study plan (HWR-791). Halden, Norway: OECD Halden Reactor Project.
36.
SkraaningG. (1998). The operator performance assessment system (OPAS) (HWR-538). Halden, Norway: OECD Halden Reactor Project.
37.
SkraaningG. (2003). Experimental control versus realism: Methodological solutions for simulator studies in complex operating environments (HPR-361). Halden, Norway: OECD Halden Reactor Project.
38.
SkraaningG. (2016). A reanalysis of the work practice experiments in HAMMLAB (2009-2014) (HWR-1194). Halden, Norway: OECD Halden Reactor Project.
39.
SkraaningG.EitrheimM. H.LauN.NihlwingC.HurlenL.KarlssonT. (2010). Coping with automation in future plants: Results from the 2009 HAMMLAB experiment (HWR-937). Halden, Norway: OECD Halden Reactor Project.
40.
SkraaningG.Jr.JamiesonG. A. (2017). Twenty years of HRP research on human-automation interaction: Insights on automation transparency and levels of automation (HPR-387). Halden, Norway: OECD Halden Reactor Project.
StowersK.KasdaglisN.NewtonO.LakhmaniS.WohleberR.ChenJ. (2016, September). Intelligent agent transparency: The design and evaluation of an interface to facilitate human and intelligent agent collaboration. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 60, No. 1, pp. 1706–1710). Los Angeles, CA: SAGE.
43.
StrandS. (2001). Trust and automation: The influence of automation malfunction and system feedback on operator trust (HWR-643). Halden, Norway: OECD Halden Reactor Project.
44.
TaylorR. M. (1990). Situational Awareness Rating Technique (SART): The development of a tool for aircrew systems design. In AGARD Conference Proceedings No 478: Situational Awareness in Aerospace Operations. Aerospace Medical Panel Symposium. Neuilly sur-Seine, France: AGARD/NATO.
45.
TrapsilawatiF.WickensC. F.ChenH.XuX. (2017). Transparency and conflict resolution automation reliability in air traffic control. In TsangP.VidulichM.FlachJ. (Eds.), Proceedings of the International Symposium on Aviation Psychology (pp. 419–424). Dayton, OH: Wright State University.
46.
WickensC. (2018). Automation stages & levels, 20 years after. Journal of Cognitive Engineering and Decision Making, 12(1), 35–41.
47.
WrightJ. L.ChenJ. Y.BarnesM. J.HancockP. A. (2016). Agent reasoning transparency’s effect on operator workload. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 60, No. 1, pp. 249–253). Los Angeles, CA: SAGE.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
