In this article, we propose a method to recognize human emotions based on user context and brain signals. We evaluated the method through an experiment during which individuals performed tasks using a simulator for electrical power systems operator training. We collected user context through log data retrieval and brain signals using an Electroencephalography (EEG) portable monitor. The experimental results demonstrated that the method could be successfully applied to recognize the emotional states based on EEG signals and user context.
JiangZ., LuL., HuangX., TanC., Design of wearable home health care system with emotion recognition function, in 2011 International Conference on Electrical and Control Engineering (2011), pp. 2995–2998.
2.
Alonso-MartínF., MalfazM., SequeiraJ., GorostizaJ.F., SalichsM.A., A multimodal emotion detection system during human–robot interaction, Sensors13(11) (2013), 15549–15581.
3.
RachedT., TorresF., AguiarY., VieiraM.F.Q., PerkusichA., Fatigue and emotions driven products: Assessing safety and performance using mobile devices, in 2014 IEEE International Conference on Consumer Electronics (ICCE) (2014), pp. 496–497.
4.
KhanM.M., WardR.D., InglebyM., Toward use of facial thermal features in dynamic assessment of affect and arousal level, IEEE Transactions on Affective Computing8(3) (2017), 412–425.
5.
MencattiniA., MartinelliE., RingevalF., SchullerB., NataleC.D., Continuous estimation of emotions in speech by dynamic cooperative speaker models, IEEE Transactions on Affective Computing8(3) (2017), 314–327.
6.
YangZ., NarayananS.S., Modeling dynamics of expressive body gestures in dyadic interactions, IEEE Transactions on Affective Computing8(3) (2017), 369–381.
7.
Arroyo-PalaciosJ., SlaterM., Dancing with physio: A mobile game with physiologically aware virtual humans, IEEE Transactions on Affective Computing7(4) (2016), 326–336.
8.
CalvoR.A., D’MelloS., Affect detection: An interdisciplinary review of models, methods, and their applications, IEEE Transactions on Affective Computing1(1) (2010), 18–37.
9.
ParkC., RyuJ., SohnJ., ChoH., An emotion expression system for the emotional robot, in 2007 IEEE International Symposium on Consumer Electronics (2007), 1–6.
10.
LiuC., RaniP., SarkarN., An empirical study of machine learning techniques for affect recognition in human-robot interaction, in 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems (2005), 2662–2667.
11.
LiuC., ConnK., SarkarN., StoneW., Physiology-based affect recognition for computer-assisted intervention of children with autism spectrum disorder, International Journal of Human-Computer Studies66(9) (2008), 662–677.
12.
ShahabiH., MoghimiS., Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity, Computers in Human Behavior58 (2016), 231–239.
13.
HuangH., XieQ., PanJ., HeY., WenZ., YuR., LiY., An eeg-based brain computer interface for emotion recognition and its application in patients with disorder of consciousness, IEEE Transactions on Affective Computing (2019), 1.
Majid MehmoodR.,
DuR. and
LeeH.J., Optimal feature selection and deep learning ensembles method for emotion recognition from human brain eeg sensors, IEEE Access5 (2017), 14797–14806.
16.
UllahH., UzairM., MahmoodA., UllahM., KhanS.D., CheikhF.A., Internal emotion classification using eeg signal with sparse discriminative ensemble, IEEE Access7 (2019), 40144–40153.
17.
KostiR., AlvarezJ.M., RecasensA., LapedrizaA., Emotion recognition in context, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), pp. 1960–1968.
18.
AbowdG.D., DeyA.K., BrownP.J., DaviesN., SmithM., StegglesP., Towards a better understanding of context and context-awareness, in Handheld and Ubiquitous Computing,
GellersenH.-W., Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 1999, pp. 304–307.
19.
BrownP.J., BoveyJ.D., ChenX., Context-aware applications: from the laboratory to the marketplace, IEEE Personal Communications4(5) (1997), 58–64.
20.
DeyA.K., Context-aware computing: The cyberdesk project, in AAAI 1998 Spring Symposium on Intelligent Environments.Palo Alto: AAAI Press., 1998, pp. 51–54.
21.
WolpawJ.R., BirbaumerN., McFarlandD.J., PfurtschellerG., VaughanT.M., Brain–computer interfaces for communication and control, Clinical Neurophysiology113(6) (2002), 767–791.
22.
GilR., Virgili-GomáJ., GarcíaR., MasonC., Emotions ontology for collaborative modelling and learning of emotional responses, Computers in Human Behavior51 (2015), 610–617, computing for Human Learning, Behaviour and Collaboration in the Social and Mobile Networks Era.
23.
WrzesienM., RodríguezA., ReyB., AlcañizM., BañosR.M., VaraM.D., How the physical similarity of avatars can influence the learning of emotion regulation strategies in teenagers, Computers in Human Behavior43 (2015), 101–111.
24.
VourvopoulosA., LiarokapisF., Evaluation of commercial brain–computer interfaces in real and virtual world environment: A pilot study, Computers and Electrical Engineering40(2) (2014), 714–729.
25.
McKenzieE.D., LimA.S.P., LeungE.C.W., ColeA.J., LamA.D., EloyanA., NirolaD.K., TsheringL., ThibertR., GarciaR.Z., BuiE., DekiS., LeeL., ClarkS.J., CohenJ.M., MantiaJ., BrizziK.T., SoretsT.R., WahlsterS., BorzelloM., StopczynskiA., CashS.S., MateenF.J., Validation of a smartphone-based EEG among people with epilepsy: A prospective study, Scientific Reports7 (2017), 45 567 EP – Apr 2017.
26.
PoulsenA.T., KamronnS., DmochowskiJ., ParraL.C., HansenL.K., Eeg in the classroom: Synchronised neural recordings during video presentation, Scientific Reports7 (2017), ,43 916 EP –, Mar 2017.
27.
BorghiniG., AricaP., Di FlumeriG.,
CartocciG.,
ColosimoA.,
BonelliS.,
GolfettiA.,
ImbertJ.P.,
GrangerG.,
BenhaceneR.,
PozziS. and
BabiloniF., Eeg-based cognitive control behaviour assessment: an ecological study with professional air traffic controllers, Scientific Reports7(1) (2017), 547.
28.
KrigolsonO.E., WilliamsC.C., NortonA., HassallC.D., ColinoF.L., Choosing muse: Validation of a lowcost, portable EEG system for ERP research, Front Neurosci11 (2017), 109. 28344546[pmid].
29.
LiuY., SubramaniamS.C.H., SourinaO., LiewS.H.P., KrishnanG., KonovessisD., AngH.E., EEG-based mental workload and stress recognition of crew members in maritime virtual simulator: A case study, in 2017 International Conference on Cyberworlds (CW) (2017), pp. 64–71.
30.
LimW.L., LiuY., Harihara SubramaniamS.C.,
LiewS.H.P.,
KrishnanG.,
SourinaO.,
KonovessisD.,
AngH.E. and
WangL., EEG-based mental workload and stress monitoring of crew members in maritime virtual simulator, in Transactions on Computational Science XXXII: Special Issue on Cybersecurity and Biometrics,
GavrilovaM.L.,
TanC.K. , and
SourinA., Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2018, pp. 15–28.
31.
PhamT.D., TranD., Emotion recognition using the emotiv EPOC device, in Neural Information Processing,
HuangT.,
ZengZ.,
LiC., and
LeungC.S., Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 394–399.
32.
SchlöglA., BrunnerC., Biosig: A free and open source software library for bci research, Computer41(10) (2008), 44–50.
33.
MateoJ., TorresA., SoriaC., SantosJ., A method for removing noise from continuous brain signal recordings, Computers and Electrical Engineering39(5) (2013), 1561–1570.
34.
JenkeR., PeerA., BussM., Feature extraction and selection for emotion recognition from EEG, IEEE Transactions on Affective Computing5(3) (2014), 327–339.
35.
BurrusC.S., GopinathR.A., GuoH., Introduction to wavelets and wavelet transforms: a primer. Upper Saddle River, N.J. Prentice Hall, 1998, includes bibliographical references (pages 224-245) and index.
36.
AulaA., SurakkaV., Auditory emotional feedback facilitates human-computer interaction, in People and Computers XVI - Memorable Yet Invisible,
FaulknerX.,
FinlayJ., and
DétienneF., Eds. London: Springer London, (2002), pp. 337–349
.
37.
CairnsP., PandabP., PowerC., The influence of emotion on number entry errors, in Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems, ser. CHI ’14. New York, NY, USA: ACM, (2014), pp. 2293–2296.
38.
ConatiC., MaclarenH., Modeling user affect from causes and effects, in User Modeling, Adaptation, and Personalization,
HoubenG.-J.,
McCallaG.,
PianesiF., and
ZancanaroM., Eds. Berlin, Heidelberg: Springer Heidelberg, (2009), pp. 4–15
.
39.
KimJ.-D., SonJ., BaikD.-K., CA5W1HOnto: Ontological context-aware model based on 5W1H, International Journal of Distributed Sensor Networks8(3) (2012), 247346.
40.
AguiarY., MercantiniC.S.J.M., VieiraC.M., GalyE., Extending a user observation protocol to account for psychological traits, in IADIS Multi Conference on Computer Science and Information Systems (2011), pp. 105–112.
41.
NettoA.V.S., VieiraM.V., Virtual reality training environment a proposed architecture, in 2010 IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurement Systems (2010), pp. 77–81.
42.
FilhoF.T., VieiraM.d.F.Q., An ontology-driven framework to support scenario representation in a 3d operator training simulator, in 2014 4th International Conference On Simulation And Modeling Methodologies, Technologies And Applications (SIMULTECH) (2014), pp. 298–303.
43.
BatesM., LemaE., The D2 test of attention: Construct validity and extensions in scoring techniques, Journal of the International Neuropsychological Society10(3) (2004), 392–400.
44.
ChongD.J., SahlemG.L., BazilC.W., CHAPTER 8 - introduction to electroencephalography, in Review of Sleep Medicine (Second Edition), second edition ed.,
BarkoukisT. J. and
AvidanA.Y., Eds. Philadelphia: Butterworth-Heinemann, (2007), pp. 105–141
.
45.
PetieauM., HoellingerT., CheronG., DutoitT., Performance of the emotiv epoc headset for p300-based applications, BioMedical Engineering OnLine12(56), 2013.
46.
AfifiA.K., BergmanR.A., Functional Neuroanatomy: Text and Atlas, 2nd ed. McGraw Hill Professional, 2005.
47.
PhanK., WagerT., TaylorS.F., LiberzonI., Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in pet and fmri, NeuroImage16(2) (2002), 331–348.
48.
FusterJ.M., The prefrontal cortex—an update: Time is of the essence, Neuron30(2) (2001), 319–333.
49.
FusterJ.M., The Prefrontal Cortex, 5th ed. London: Academic Press, 2015.
50.
Matlab optimization toolbox, 2015, the MathWorks, Natick, MA, USA.
51.
MurugappanM., NagarajanR., YaacobS., Appraising human emotions using time frequency analysis based EEG alpha band features, in 2009 Innovative Technologies in Intelligent Systems and Industrial Applications (2009), pp. 70–75.
52.
DaubechiesI., Orthonormal bases of compactly supported wavelets, Communications on Pure and Applied Mathematics41(7) (1988), 909–996.
53.
SubasiA., EEG signal classification using wavelet feature extraction and a mixture of expert model, Expert Systems with Applications32(4) (2007), 1084–1093.
54.
WangX.-W., NieD., LuB.-L., Emotional state classification from EEG data using machine learning approach, Neurocomputing129 (2014), 94–106.
55.
JahankhaniP., KodogiannisV., RevettK., EEG signal classification using wavelet feature extraction and neural networks, in IEEE John Vincent Atanasoff 2006 International Symposium on Modern Computing (JVA’06) (2006), pp. 120–124.
56.
SohaibA.T., QureshiS., HagelbäckJ., HilbornO., JerčićP., Evaluating classifiers for emotion recognition using eeg, in Foundations of Augmented Cognition,
SchmorrowD. D.
and
FidopiastisC.M., Eds. Berlin Heidelberg: Springer Berlin Heidelberg, (2013), pp. 492–501
.
57.
LotteF., CongedoM., LécuyerA., LamarcheF., ArnaldiB., A review of classification algorithms for EEG-based brain–computer interfaces, Journal of Neural Engineering4(2) (2007), R1.
58.
hoC.K., SasakiM., EEG data classification with several mental tasks, in Man and Cybernetics6 (2002), 4.
59.
ChenG., HouR., A new machine double-layer learning method and its application in non-linear time series forecasting, in 2007 International Conference on Mechatronics and Automation (2007), pp. 795–799.
60.
ChenJ.X., ZhangP.W., MaoZ.J., HuangY.F., JiangD.M., ZhangY.N., Accurate eeg-based emotion recognition on combined features using deep convolutional neural networks, IEEE Access7 (2019), 44317–44328.
61.
MeiH., XuX., Eeg-based emotion classification using convolutional neural network, in 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC) (2017), pp. 130–135.
62.
Meza-KuboV., MoránA.L., CarrilloI., GalindoG., García-CansecoE., Assessing the user experience of older adults using a neural network trained to recognize emotions from brain signals, Journal of Biomedical Informatics62 (2016), 202–209.
63.
RoyY., BanvilleH.J., AlbuquerqueI., GramfortA., FalkT.H., FaubertJ., Deep learning-based electroencephalography analysis: a systematic review, Journal of Neural Engineering16(3) (2019), 051001.
64.
CraikA., HeY., Contreras-VidalJ.L., Deep learning for electroencephalogram (eeg) classification tasks: a review, Journal of Neural Engineering16(5) (2019), 031001.
65.
SinghG., JatiA., KhasnobishA., BhattacharyyaS., KonarA., TibarewalaD.N., JanarthananR., Negative emotion recognition from stimulated EEG signals, in Computing Communication Networking Technologies (ICCCNT), 2012 Third International Conference on (2012), pp. 1–8.
66.
LevineD., Applied Statistics for Engineers and Scientists: Using Microsoft Excel and MINITAB.Pearson Education, Limited, 2001.
