Video visual analytics is the research field that addresses scalable and reliable analysis of video data. The vast amount of video data in typical analysis tasks renders manual analysis by watching the video data impractical. However, automatic evaluation of video material is not reliable enough, especially when it comes to semantic abstraction from the video signal. In this article, we describe the video visual analytics method that combines the complementary strengths of human recognition and machine processing. After inspecting the challenges of scalable video analysis, we derive the main components of visual analytics for video data. Based on these components, we present our video visual analytics system that has its origins in our IEEE VAST Challenge 2009 participation.
GrinsteinGScholtzJWhitingM. VAST 2009 challenge: an insider threat. In: IEEE symposium on visual analytics science and technology, 2009 (VAST 2009), Salt Lake City, Utah, USA, October, pp. 243–244. IEEE.
6.
ThomasJJCookKA. Illuminating the path: the research and development agenda for visual analytics. IEEE Computer Society, 2005.
7.
PirolliPCardS. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: Proceedings of international conference on intelligence analysis, McLean, VA, USA, 3–5 May 2005.
8.
KeimDAMansmannFSchneidewindJ. Visual analytics: scope and challenges. In: SimoffSJBöhlenMHMazeikaA (eds) Visual data mining (vol. 4404 of Lecture notes in computer science). Berlin and Heidelberg: Springer-Verlag, 2008, pp. 76–90.
9.
XiongZZhouXSTianQ. Semantic retrieval of video—review of research on video retrieval in meetings, movies and broadcast news, and sports. IEEE Signal Proc Mag2006; 23(2): 18–27.
10.
GoodwinCGoodwinMH. Seeing as situated activity: formulating planes. In: Yrjö EngeströmDM (ed.) Cognition and communication at work. New York: Cambridge University Press, 1996, pp. 61–95.
11.
LomellHM. Targeting the unwanted: video surveillance and categorical exclusion in Oslo, Norway. Surveill Soc2002; 2(2/3): 346–360.
12.
FrancoisARJNevatiaRHobbsJ. VERL: an ontology framework for representing and annotating video events. IEEE Multimedia2005; 12(4): 76–86.
SvenssonMHeathCLuffP. Monitoring practice event detection and system design. In: VelastinSARemagninoP (eds) Intelligent distributed video surveillance systems (vol. 5 of IEEE professional applications of computing series). London: The Institution of Engineering and Technology, 2006, pp. 1–30.
15.
KevalHU. Effective, design, configuration, and use of digital CCTV. PhD Thesis, University CollegeLondon, UK, 2009.
16.
HöferlinMHöferlinBWeiskopfD. Uncertainty-aware video visual analytics of tracked moving objects. JOSIS: J Spat Inf Sci2011; 2: 87–117.
17.
NusimowA. Intelligent video for homeland security applications. In: IEEE conference on technologies for homeland security, Woburn, MA, 16–17 May 2007, pp. 139–144. IEEE.
18.
IP Video Market. Video surveillance book 2012 (Electronic). Honolulu, HI: IP Video Market, 2012.
19.
WangHKlaserASchmidC. Action recognition by dense trajectories. In: IEEE conference on computer vision and pattern recognition (CVPR), Providence, RI, 20–25 June 2011, pp. 3169–3176. IEEE.
20.
MarszalekMLaptevISchmidC. Actions in context. In: IEEE conference on computer vision and pattern recognition (CVPR 2009), Miami, FL, 20–25 June 2009, pp. 2929–2936. IEEE.
21.
KeimDAMansmannFThomasJ. Visual analytics: how much visualization and how much analytics?ACM SIGKDD Explor Newsl2010; 11(2): 5–8.
22.
BertiniELalanneD. Investigating and reflecting on the integration of automatic data analysis and visualization in knowledge discovery. ACM SIGKDD Explor Newsl2010; 11(2): 9–18.
23.
TukeyJW. Exploratory data analysis. Reading, MA: Addison-Wesley, 1977.
24.
FayyadUPiatetsky-ShapiroGSmythP. From data mining to knowledge discovery in databases. AI Mag1996; 17(3): 37–54.
25.
WurmanRS. Information anxiety: what to do when information doesn’t tell you what you want to know. New York: Bantam Books, 1989.
26.
EndsleyMR. Designing for situation awareness in complex systems. In: Proceedings of the second international workshop on symbiosis of humans, artifacts and environment, Kyoto, Japan, 2001.
27.
EndsleyMR. Toward a theory of situation awareness in dynamic systems. Hum Fac Erg Soc P1995; 37(1): 32–64.
28.
RensinkRAO’ReganJKClarkJJ. To see or not to see: the need for attention to perceive changes in scenes. Psychol Sci1997; 8(5): 368–373.
GreenMWRenoJFisherRC. The appropriate and effective use of security technologies in US schools: a guide for schools and law enforcement agencies aeries. Research Report, National Institute of Justice, Washington, DC, September1999.
31.
WorringMSnoekCGMde RooijO. The MediaMill semantic video search engine. In: IEEE conference on acoustics, speech and signal processing (ICASSP 2007), Honolulu, HI, 15–20 April 2007, vol. 4, pp. IV-1213–IV-1216. IEEE.
32.
SnoekCvan de SandeKde RooijO. The MediaMill TRECVID 2009 Semantic Video Search Engine In: TREC Video Retrieval Evaluation 2009, Gaithersburg, MD, 16-17 November 2009. National Institute of Standards and Technology (NIST).
33.
ChristelMG. Supporting video library exploratory search: when storyboards are not enough. In: Proceedings of the 2008 international conference on content-based image and video retrieval (CIVR’08), Niagara Falls, Canada, 7–9 July 2008, pp. 447–456. New York: ACM.
34.
HauptmannAGChristelMGYanR. Video retrieval based on semantic concepts. Proc IEEE2008; 96(4): 602–622.
35.
LuoHFanJYangJ. Exploring large-scale video news via interactive visualization. In: IEEE symposium on visual analytics science and technology, Baltimore, MD, 31 October–2 November 2006, pp. 75–82. IEEE.
36.
LuoHFanJSatohS. Integrating multi-modal content analysis and hyperbolic visualization for large-scale news video retrieval and exploration. Signal Processing: Image Communication2008; 23(7): 538–553.
37.
AndrienkoGAndrienkoNDemsarU. Space, time and visual analytics. Int J Geogr Inf Sci2010; 24(10): 1577–1600.
38.
BotchenRPBachthalerSSchickF. Action-based multifield video visualization. IEEE T Vis Comput Gr2008; 14(4): 885–899.
39.
KraakMJ. The space-time cube revisited from a geovisualization perspective. In: Proceedings of the 21st international cartographic conference (ICC’03): cartographic renaissance, Durban, South Africa, 10–16 August 2003, pp. 1988–1996.
40.
AndrienkoNAndrienkoGGatalskyP. Exploratory spatio-temporal visualization: an analytical review. J Visual Lang Comput2003; 14(6): 503–541.
41.
WadeNJSwanstonMT. Visual perception: an introduction. 2nd ed.Hove, East Sussex: Psychology Press Ltd, 2001.
DavisFWSimonettDS. GIS and remote sensing. In: MaguireDJGoodchildMFRhindDW (eds) Geographical information systems. London: Longman House, 1991, pp. 191–213.
44.
LongleyPAGoodchildMFMaguireDJ. Geographical information systems and science. New York: John Wiley & Sons Ltd, 2005.
45.
KruegleH. CCTV surveillance: analog and digital video practices and technology. 2nd ed.Burlington, MA, USA: Elsevier Butterworth-Heinemann, 2007.
46.
KevalHSasseMA. To catch a thief—you need at least 8 frames per second: the impact of frame rates on user performance in a CCTV detection task. In: Proceedings of the ACM international conference on multimedia, Vancouver, British Columbia, Canada, 27-31 October 2008, pp. 941–944. ACM.
47.
KevalHUSasseMA. Can we ID from CCTV: image quality in digital CCTV and face identification performance. Soc Photo-Opt Instru2008; 6982.
48.
AndrienkoGAndrienkoNKopanakisI. Visual analytics methods for movement data. In: Fosca GiannottiDP (ed.) Mobility, data mining and privacy: geographic knowledge discovery. Berlin and Heidelberg: Springer-Verlag, 2008, pp. 375–408.
49.
KuhnASenstTKellerI. A Lagrangian framework for video analytics. In: IEEE 14th international workshop on multimedia signal processing (MMSP’2012), Banff, Canada, 17–19 September 2012, pp. 387–392. IEEE.
50.
MansmannFFischerFKeimDA. Dynamic visual analytics—facing the real-time challenge. In: DillJEarnshawRAKasikDJ. (eds) Expanding the frontiers of visual analytics and visualization. London: Springer, 2012, pp. 69–80.
51.
AignerWMikschSSchumannH. Visualization of time-oriented data. New York: Springer-Verlag, 2011.
52.
BorgoRChenMGrundyE. State of the art report on video-based graphics and video visualization. Comput Graph Forum2012; 31: 2450–2477.
53.
BoschHHeinrichJHöferlinB. Innovative filtering techniques and customized analytics tools. In: IEEE symposium on visual analytics science and technology (VAST’09), Atlantic City, NJ, 12–13 October 2009, pp. 269–270. IEEE Computer Society.
54.
BabcockBBabuSDatarM. Models and issues in data stream systems. In: Proceedings of the 21st ACM SIGMOD-SIGACT-SIGART symposium on principles of database systems, Madison, Wisconsin, 3-6 June 2002, pp. 1–16. NY, USA: ACM.
55.
BroiloMPiottoNBoatoG. Object trajectory analysis in video indexing and retrieval applications. In: SchonfeldDShanCTaoD. (eds) Video search and mining, vol. 287. Berlin Heidelberg: Springer, 2010, pp. 3–32.
56.
HöferlinBNetzelRHöferlinM. Inter-active learning of ad-hoc classifiers for video visual analytics. In: IEEE conference on visual analytics science and technology (VAST), Seattle, WA, 14–19 October 2012, pp. 23–32.
57.
PekerKDivakaranA. Adaptive fast playback-based video skimming using a compressed-domain visual complexity measure. In: Proceedings of the IEEE international conference on multimedia and expo (ICME’04), Taipei, Taiwan, 27–30 June 2004, pp. 2055–2058. IEEE.
HöferlinBPflügerHHöferlinM. Learning a visual attention model for adaptive fast-forward in video surveillance. In: Proceedings of the international conference on pattern recognition applications and methods (ICPRAM), Vilamoura, Algarve, Portugal, 6-8 February 2012, pp. 25–32. Institute for Systems and Technologies of Information Control and Communication (INSTICC).
60.
RobertsJC. State of the art: coordinated & multiple views in exploratory visualization. In: IEEE fifth international conference on coordinated and multiple views in exploratory visualization (CMV’07), Zurich, Switzerland, 2 July 2007, pp. 61–71. IEEE.
61.
HöferlinBHöferlinMRäuchleJ. Visual analytics of mobile data. In: Proceedings of the Nokia mobile data challenge 2012 workshop, Newcastle, UK, 18-19 June 2012.
62.
HöferlinMHöferlinBWeiskopfD. Interactive schematic summaries for faceted exploration of surveillance video. IEEE T Multimedia2012; PP(99): 1.
63.
HöferlinMKurzhalsKHöferlinB. Evaluation of fast-forward video visualization. IEEE T Vis Comput Gr2012; 18: 2095–2103.
HöferlinBHöferlinMGoloubetsB. Auditory support for situation awareness in video surveillance. In: Proceedings of the international conference on auditory display (ICAD), Atlanta, Georgia, 18-21 June 2012.
66.
HöferlinBHöferlinMRaschkeM. Interactive auditory display to support situational awareness in video surveillance. In: Proceedings of the international conference on auditory display (ICAD), Budapest, Hungary, 20-24 June 2011.
67.
EllisGDixA. An explorative analysis of user evaluation studies in information visualisation. In: Proceedings of the 2006 AVI workshop on beyond time and errors: novel evaluation methods for information visualization, Venezia, Italy, 23-26 May 2006, pp. 15–21. NY, USA: ACM.
