To shed more light on data explorers dealing with complex information visualizations in real-world scenarios, new methodologies and models are needed which overcome existing explanatory gaps. Therefore, a novel model to analyze users’ errors and insights is outlined that is derived from Rasmussen’s model on different levels of cognitive processing, and integrates explorers’ skills, schemes, and knowledge (skill–rule–knowledge model). After locating this model in the landscape of theories for visual analytics, the main building blocks of the model, where three cognitive processing levels are interlinked, are described in detail. A case study illustrates how the cognitive processing efforts can be identified from triangulated eye-tracking and think-aloud data. Finally, the model’s applicability, challenges in measurement, and future research options are discussed.
LamHIsenbergPIsenbergT. BELIV’14—proceedings of the fifth workshop on beyond time and errors: novel evaluation methods for visualization. New York: ACM, 2014, http://dx.doi.org/10.1145/2669557
2.
BertiniEPlaisantCSantucciG. BELIV’06: beyond time and errors; novel evaluation methods for information visualization. Interactions2007; 14(3): 59–60, http://doi.acm.org/10.1145/1242421.1242460
3.
EllisGDixA. An explorative analysis of user evaluation studies in information visualization. In: Proceedings of the 2006 AVI workshop on beyond time and errors (BELIV’06), 2006, pp. 1–7. New York: ACM, http://doi.acm.org/10.1145/1168149.1168152
4.
AndrewsK. Evaluating information visualizations. In: Proceedings of the 2006 AVI workshop on beyond time and errors: novel evaluation methods for information visualization (BELIV’06), 2006, pp. 1–5. New York: ACM, http://doi.acm.org/10.1145/1168149.1168151
5.
PlaisantC. The challenge of information visualization evaluation. In: Proceedings of the working conference on advanced visual interfaces (AVI’04), 2004, pp. 109–116. New York: ACM, http://doi.acm.org/10.1145/989863.989880
SmucM. Just the other side of the coin? From error- to insight-analysis. In: Proceedings of the fifth workshop on beyond time and errors: novel evaluation methods for visualization (BELIV’14), Paris, 9–14 November 2014, pp. 35–40. New York: ACM, http://doi.acm.org/10.1145/2669557.2669570
8.
ThomasJJCookKA. Illuminating the path: the research and development agenda for visual analytics. Los Alamitos, CA: IEEE CS Press, 2005.
9.
CarpenterPAShahP. A model of the perceptual and conceptual processes in graph comprehension. J Exp Psychol: Appl1998; 4(2): 75–100.
10.
KosslynS. Understanding charts and graphs. Appl Cognitive Psych1989; 3: 185–225.
11.
ToryMMollerT. Human factors in visualization research. IEEE T Vis Comput Gr2004; 10: 72–84.
12.
TraftonJGKirschenbaumSSTsuiTL. Turning pictures into numbers: extracting and generating information from complex visualizations. Int J Hum: Comput St2000; 53(5): 827–850.
13.
PirolliPCardS. The sensemaking process and leverage points for analyst technology as identified through CTA. In: Proceedings of the 2005 international conference on intelligence analysis, McLean, VA, 3–5 May 2005, pp. 2–4.
14.
SpringmeyerRBlattnerMMaxNL. A characterization of the scientific data analysis process. In: Proceedings of the 3rd IEEE conference on visualization’92, Boston, MA, 19–23 October 1992, pp. 235–242. New York: IEEE.
15.
ReasonJ. Human error. New York: Cambridge University Press, 1990.
16.
PohlMSmucMMayrE. The user puzzle—explaining the interaction with visual analytics systems. IEEE T Vis Comput Gr2012; 18: 2908–2916.
17.
RasmussenJ. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE T Syst Man Cyb1983; 13: 257–266.
18.
RasmussenJPejtersenAMSchmidtK. Taxonomy for cognitive work analysis. Report, Risø-M-2871. Roskilde: Risø National Laboratory, September, 1990.
19.
SmucMMayrELammarschT. To score or not to score? Tripling insights for participatory design. IEEE Comput Graph2009; 29(3): 29–38.
20.
SaraiyaPBNorthCDucaK. An insight-based methodology for evaluating bioinformatics visualizations. IEEE T Vis Comput Gr2005; 11: 443–456.
AkersDSimpsonMJeffriesR. Undo and erase events as indicators of usability problems. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’09), 2009, pp. 659–668. New York: ACM, http://doi.acm.org/10.1145/1518701.1518804
24.
RicheN. Beyond system logging: human logging for evaluating information visualization. In: Proceedings of the BELIV’10 workshop (BELIV’10)—workshop at ACM SIGCHI, Atlanta, GA, 10–11 April 2010.
25.
FlanaganJC. The critical incident technique. Psychol Bull1954; 51(4): 327–358.
HessEHPoltJM. Pupil size in relation to mental activity during simple problem-solving. Science1964; 143(3611): 1190–1192.
28.
AndreassiJL. Psychophysiology: human behavior and physiological response. Hillsdale, NJ: Lawrence Erlbaum Associates, 1995.
29.
BeattyJLucero-WagonerB. The pupillary system. In: CacioppoJTTassinaryLGBerntsonGG (eds) Handbook of psychophysiology. 2nd ed.New York: Cambridge University Press, 2000, pp. 142–162.
30.
FerdousNJianuR. Using pupil size as an indicator for task difficulty in data visualization. In: IEEE VIS, poster program, Paris, 9–14 November 2014.
31.
JustMACarpenterPA. Eye fixations and cognitive processes. Cognitive Psychol1976; 8(4): 441–480.
32.
VelichkovskyBMChallisBHPomplunM. Working memory and work with memory: visuospatial and further components of processing. Z Exp Psychol1995; 42: 672–701.
33.
VelichkovskyBM. From levels of processing to stratification of cognition: converging evidence from three domains of research. In: ChallisBHVelichkovskyBM (eds) Stratification in cognition and consciousness. Amsterdam: John Benjamins, 1999, pp. 203–235.
34.
SmucMFedericoPWindhagerF. How do you connect moving dots? Insights from user studies on dynamic network visualizations. In: HuangW (ed.) Handbook of human centric visualization. New York: Springer, 2014, pp. 623–650.
35.
SmucMMayrERiskuH. Is your user hunting or gathering insights? Identifying insight drivers across domains. In: Proceedings of the BELIV’10 workshop (BELIV’10), Atlanta, GA, 10–15 April 2010, pp. 49–54. New York: ACM.
36.
MayrESmucMRiskuH. Many roads lead to Rome: mapping users’ problem solving strategies. In: Proceedings of the BELIV’10 workshop (BELIV’10), Atlanta, GA, 10–15 April 2010, pp. 8–15. New York: ACM.
37.
SmucMMayrELammarschT. Visualizations at first sight: do insights require training? In: HolzingerA (ed.) HCI and usability for education and work. Berlin: Springer, 2008, pp. 261–280.
38.
TraftonJGTrickettSB. A new model of graph and visualization usage. In: The proceedings of the 23rd annual conference of the CSS, University of Edinburgh, Edinburgh, 1–4 August 2001, pp. 1048–1053. Mahwah, NJ: Lawrence Erlbaum Associates.
