The aim of this survey is to provide an overview of the various components of “computer aided affective learning systems.” The research is classified into 3 main scientific areas that are integral parts of the development of these kinds of systems. The three main scientific areas are: I) emotions and their connection to learning; ii) affect recognition; and iii) emotional instruction and design. Affective learning instructional technology is a new, multi-disciplinary research area, which has been developed during the last decade. This article depicts the development of the core relevant areas and describes the concerns.
Get full access to this article
View all access options for this article.
References
1.
AlderG. S. (2007). Examining the relationship between feedback and performance in a monitored environment: A clarification and extension of feedback intervention theory. Journal of High Technology Management Research, 17(2), 157–174.
2.
AndersonN. H. (1982). Methods of information integration theory.London: Academic Press.
3.
ArgyleM. (1988). Bodily communication (2nd ed.). New York: Methuen.
4.
ArkW.DryerD., & LuD. (1999). The emotion mouse. In BullingerH. J. & ZieglerJ. (Eds.), Human-computer interaction: Ergonomics and user interfaces (pp. 818–823). London: Lawrence Erlbaum.
5.
ArmbrusterB. (1986). Schema theory and the design of content-area textbooks. Educational Psychologist, 21(4), 253–267.
AstleitnerH. (2000b). Designing instructional technology from an emotional perspective. Journal of Research on Computing in Education, 32(4), 497–510.
8.
AstleitnerH. (2001). Designing emotionally sound instruction—An empirical validation of the FEASP-approach. Journal of Instructional Psychology, 28(4), 209–219.
9.
BamidisP. D.PapadelisC.Kourtidou-PapadeliC., & VivasA. (2004). Affective computing in the era of contemporary neurophysiology and health informatics. Interacting with Computers, 16(4), 715–721.
10.
BanseR., & ShererK. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70(3), 614–636.
11.
BartneckC. (2002). Integrating the OCC model of emotions in embodied characters. In Proceedings of the workshop on virtual conversational characters: Applications, methods, and research challenges, Melbourne.
12.
BecharaA.DamasioH.TranelD., & DamasioA. (1997). Deciding advantageously before knowing the advantageous strategy. Science, 275(5304), 1293–1295.
13.
BestR. (2003). Struggling with the spiritual in education. In Tenth international conference education spirituality and the whole child conference, University of Surrey Roehampton, London.
14.
BoshL. T. (2000). Emotions: What is possible in the ASR framework? In Proceedings of the International Speech Communication Association workshop on speech and emotion (ISCA 2000), Belfast, Northern Ireland.
15.
BowerG. (1992). How might emotions affect learning? In SvenakeC. & LawrenceE. (Eds.), Handbook of emotion and memory: Research and theory.Hillsdale, NJ: Erlbaum.
16.
BratmanM. E. (1990). What is intention? In CohenP.MorganJ., & PollackM. (Eds.), Intentions in communication (pp. 15–31). Cambridge, MA: MIT Press.
17.
BraveS.NassC., & HutchinsonK. (2005). Computers that care: Investigating the effects of orientation of emotion exhibited by an embodied computer agent. International Journal of Human-Computer Studies, 62(2), special issue: Subtle expressivity for characters and robots, 161–178.
18.
BurkhardtF., & SendlmeierW. (2000). Verification of acoustical correlates of emotional speech using formant-synthesis. In Proceedings of the International Speech Communication Association workshop on speech and emotion (ISCA 2000), Belfast, Northern Ireland.
19.
BurlesonW., & PicardR. W. (2004). Affective agents: Sustaining motivation to learn through failure and a state of “stuck.” In Workshop on social and emotional intelligence in learning environments, 7th conference on intelligent tutoring systems, Maceió, Brasil.
20.
CamurriA., & CoglioA. (1998). An architecture for emotional agents. IEEE MultiMedia, 5(4), 24–33.
21.
CassellJ.PelachaudC.BadlerN.SteedmanM.AchornB.BecketT. (1994). Animated conversation: Rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In Proceedings of special interest group for computer GRAPHics (SIGGRAPH 94) (pp. 413–420). New York: ACM Press.
22.
CassellJ.SullivanJ.PrevostS., & ChurchillE. (2000). Embodied conversational agents.Cambridge, MA: MIT Press.
CohenI.SebeN.ChenL.GargA., & HuangT. S. (2003). Facial expression recognition from video sequences: Temporal and static modelling. Computer Vision and Image Understanding, 91(1–2), 160–187.
25.
ConatiC. (2002). Probabilistic assessment of user's emotions in education games. Journal of Applied Artificial Intelligence, 16(7–8), special issue on managing cognition and Affect in HCI, 555–575.
26.
ConatiC., & ZhouX. (2002). Modeling students' emotions from cognitive appraisal in educational games. In CerriS. A.GouarderesG., & ParaguacuF. (Eds.), Intelligent tutoring systems (pp. 944–954). Berlin Heidelberg: Springer-Verlag.
27.
CooperB. (2006). The significance of emotion and empathy in learning with MC3. In Proceedings of the sixth IEEE international conference on advanced learning technologies (pp. 1204–1205), Kerkrade, The Netherlands.
28.
CowieR. (2003). Describing the emotional states expressed in speech. Speech Communication, 40(1–2), special issue on speech and emotion, 5–32.
29.
CraigS. D.GraesserA. C.SullinsJ., & GholsonB. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250.
30.
DamasioA. R. (1994). Descartes error: Emotion, reason and the human brain.New York: G. P. Putnam Sons.
31.
DamasioA. R. (2003). Looking for Spinoza: Joy, sorrow and the feeling brain.London: Heinemann.
32.
DamjanovicV.KravcikM., & DevedzicV. (2005). eQ: An adaptive educational hypermedia-based BDI agent system for the semantic web. In SampsonD. G. & GoodyearP. (Eds.), Fifth IEEE International Conference on Advanced Learning Technologies (ICALT 2005), pp. 421–423, Kaohsiung.
33.
DehnD. M., & Van MulderS. (2000). The impact of animated interface agents: A review of empirical research. International Journal of Human-Computer Studies, 52(1), 1–22.
34.
del SoldatoT., & du BoulayB. (1995). Implementation of motivational tactics in tutoring systems. Journal of Artificial Intelligence in Education, 6(4), 337–378.
35.
DellaertF.PolzinT., & WaibelA. (1996). Recognizing emotion in Speech. In Proceedings of international conference on spoken language processing (ICSLP '96), 1970–1973, Philadelphia, PA.
36.
Den UylM. J., & van KuilenburgH. (2005). The FaceReader: Online facial expression recognition. In Proceedings of measuring behaviour (pp. 589–590), Wageningen, The Netherlands.
37.
DieterichH.MalinowskiU.KühmeT., & Schneider-HufschmidtM. (1993). State of the art in adaptive user interfaces. In Schneider-HufschmidtM.KühmeT., & MalinowskiU. (Eds.), Adaptive user interfaces: Principles and practice (pp. 13–48). Amsterdam: North Holland Elsevier.
38.
EconomidesA. A. (2005a). Adaptive feedback evaluation. In Proceedings of the 5th WSEAS international conference on distance learning and web engineering (pp. 134–139). Corfu Island, Greece.
39.
EconomidesA. A. (2005b). Personalized feedback in CAT (Computer Adaptive Testing). WSEAS Transactions on Advances in Engineering Education, 2(3), 174–181.
40.
EconomidesA. A. (2006). Emotional feedback in CAT (Computer Adaptive Testing). International Journal of Instructional Technology & Distance Learning, 3(2) [online].
41.
EconomidesA. A., & MoridisC. N. (2008). Adaptive self-assessment trying to reduce fear. In Proceedings first international conference on advances in computer-human interaction. ACHI 2008, IEEE Press (pp. 158–163), Sainte Luce, Martinique.
42.
EkmanP. (1982). Emotion in the human face (2nd ed.). Cambridge, MA: Cambridge University Press.
43.
EkmanP. (1985). Telling lies.New York: W. W. Norton.
44.
EkmanP.(1992). An argument for basic emotions. Cognition and Emotion, 6(3), 169–200.
45.
EkmanP. (1994). Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique. Psychological Bulletin, 115(2), 268–287.
46.
EkmanP., & FriesenW. V. (1977). Manual for the facial action coding system.Palo Alto, CA: Consulting Psychologists Press.
47.
EkmanP.LevensonR. W., & FriesenW. V. (1983). Autonomic nervous system activity distinguishes among emotions. Science, 221(4616), 1208–1210.
48.
EssaI. A., & PentlandA. P. (1997). Coding, analysis, interpretation and recognition of facial expressions. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 757–763.
49.
FerdigR. E., & MishraP. (2004). Emotional responses to computers: Experiences in unfairness, anger, and spite. Journal of Educational Multimedia and Hypermedia, 13(2), 143–161.
50.
FrijdaN. (1986). The emotions.Cambridge, UK: Cambridge University Press.
51.
Glaser-ZikudaM.FussS.LaukenmannM.MetzK., & RandlerC. (2005). Promoting students' emotions and achievement—Instructional design and evaluation of the ECOLE-approach. Learning and Instruction, 15, 481–495.
GriffithsM. D., & HuntN. (1995). Computer game playing in adolescence—Prevalence and demographic indicators. Journal of Community and Applied Social Psychology, 5(3), 189–193.
54.
HoneK. (2005). Empathic agents to reduce user frustration: The effects of varying agent characteristics. Interacting with Computers, 18(2), 227–245.
55.
IsenA. M. (2003). Positive affect as a source of human strength. In AspinwallL. G. & StaudingerU. M. (Eds.), A psychology of human strengths: Fundamental questions and future directions for a positive psychology (pp. 179–195). Washington, DC: APA Books.
56.
JamesW. (1884). What is an emotion?First Published in Mind, 9, 188–205.
57.
JaquesP., & ViccariR. M. (2004). A BDI approach to infer student's emotions in an intelligent learning environment. In IBERAMIA 2004 Ibero-American Conference on Artificial Intelligence (pp. 901–911, Puebla, Mexico.
58.
JiangH.VidalJ. M., & HuhnsM. N. (2007). EBDI: An architecture for emotional agents. In Proceedings of the 6th International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS '07) (pp. 38–40). Honolulu, HI: ACM Press.
59.
KapoorA., & PicardR. W. (2005). Multimodal affect recognition in learning environments. In Proceedings of the 13th annual ACM international conference on multimedia (pp. 677–682), Hilton, Singapore.
60.
KatsionisG., & VirvouM. (2005). Adapting OCC theory for affect perception in educational software. In Proceedings of the international conference on humancomputer interaction, Las Vegas, Nevada USA, Vol. 6—Human Factors Issuesin HCI.
61.
KleinJ.MoonY., & PicardR. W. (2002). This computer responds to user frustration: Theory, design and results. Interacting with Computers, 14(2), 119–140.
62.
KleinginnaP. R., & KleinginnaA. M. (1981). A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and Emotion, 5(4), 345–379.
63.
KortB.ReillyR., & PicardR. (2001a). An affective model of interplay between emotions and learning: reengineering educational pedagogy—Building a learning companion. In OkamotoT.HartleyR.KinshukJ., & KlusP. (Eds.), Proceedings of the IEEE international conference on advanced learning technology: Issues, achievements and challenges (pp. 43–48). Madison, WI, IEEE Computer Society.
64.
KortB.ReillyR., & PicardR. W. (2001b). External representation of learning process and domain knowledge: Affective state as a determinate of its structure and function. In Proceedings of the artificial intelligence in education workshops (San Antonio, TX, AIED) (pp. 64–69).
65.
LealM. R. M. (2002). The caring relationship. In Ninth annual international conference education spirituality and the whole child, Roehampton, London.
66.
LeeC. M., & NarayananS. (2005). Towards detecting emotions in spoken dialogs. IEEE Transactions in Speech and Audio Processing, 13(2), 293–303.
67.
LeeE. J.NassC., & BraveS. (2000). Can computer-generated speech have gender? An experimental test of gender stereotype. In Proceedings of the conference on human factors in computing systems (CHI' 00), extended abstracts on human factors in computing systems (pp. 289–290), The Hague, The Netherlands.
68.
LeidelmeijerK. (1991). Emotions: An experimental approach.Tilburg: Tilburg University Press.
69.
LesterJ.ConverseS.KahlerS.BarlowT.StoneB., & BhogalR. (1997). The persona effect: Affective impact of animated pedagogical agents. In Proceedings of the conference on human factors in computing systems (CHI '97) (pp. 359–366), Atlanta.
70.
LesterJ.VoermanJ. L.TownsS. G., & CallawayC. B. (1999). Deictic believability: Coordinating gesture locomotion and speech in lifelike pedagogical agents. Applied Artificial Intelligence, 13(4), 383–414.
71.
LightA. (2004). Designing to persuade: The use of emotion in networked media. Interacting with Computers, 16(4), 729–738.
72.
LindgaardG. (1995). Human performance in fault diagnosis: Can expert systems help. Interacting with Computers, 7(3), 254–272.
73.
LindgaardG. (2004). Adventurers versus nit-pickers on affective computing. Interacting with Computers, 16(4), 723–728.
74.
LindgaardG., & TriggsT. J. (1990). Can artificial intelligence outperform real people? The potential of computerised decision aids in medical diagnosis. In KarwowskiW.GenaidyA., & AsfourS. S. (Eds.), Computer aided design: Applications in ergonomics and safety (pp. 416–422). London: Taylor & Francis.
75.
LisettiC. L., & SchianoD. J. (2000). Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect. Pragmatics and Cognition, 8(1) special issue on facial information processing, 185–235.
76.
MaturanaH. R., & VarelaA. (1992). The tree of knowledge: The biological roots of human understanding.Boston: Shambhala.
77.
MavrikisM.MaciociaA., & LeeJ. (2003). Targeting the affective state of students studying mathematics on a web-based ILE. In Young researchers track of the 11th international conference on artificial intelligence in education (AIED), Sydney, Australia.
78.
McQuigganS.LeeS., & LesterJ. (2006). Predicting user physiological response for interactive environments: An inductive approach. In Proceedings of the second conference on artificial intelligence and interactive entertainment (pp. 60–65), Marina del Rey.
79.
MotaS., & PicardR. W. (2003). Automated posture analysis for detecting learner's interest level. In Proceedings of computer vision and pattern recognition (CVPR 2003) Workshop on HCI (pp. 724–736), Madison, Wisconsin.
80.
NassC., & MoonY. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81–103.
NormanD. A. (2002). Emotion and design: Attractive things work better. Interactions Magazine, 9(4), 36–42.
83.
OatleyK. (2004). The bug in the salad: The uses of emotions in computer interfaces. Interacting with Computers, 16(4), 693–696.
84.
OliveiraE., & SarmentoL. (2003). Emotional advantage for adaptability and autonomy. In Proceedings of the second international joint conference on autonomous agents and multi-agent systems (pp. 305–312), Melbourne, Australia.
85.
OliverN.PentlandA., & BerardF. (2000). LAFTER: A real-time face and lips tracker with facial expression recognition. Pattern Recognition, 33(8), 1369–1382.
86.
OrtonyA.CloreG. L., & CollinsA. (1988). The cognitive structure of emotions.Cambridge, UK: Cambridge University Press.
87.
OudeyerP.-Y. (2003). The production and recognition of emotions in speech: Features and algorithms. International Journal of Human-Computer Studies, 59(1–2), 157–183.
88.
OviattS. (2003). User-modeling and evaluation of multimodal interfaces. In Proceedings of the IEEE, special issue on human-computer multimodal interface, 91(9), 1457–1468.
89.
PanticM., & RothkrantzL. J. M. (2003). Toward an affect-sensitive multimodal humancomputer interaction. In Proceedings of the IEEE, special issue on human-computer multimodal interface, 91(9), 1370–1390.
90.
PartalaT., & SurakkaV. (2003). Pupil size as an indication of affective processing. International Journal of Human-Computer Studies, 59(1–2), 185–198.
91.
PartalaT., & SurakkaV. (2004). The effects of affective interventions in human-computer interaction. Interacting with Computers, 16(2), 295–309.
92.
PartalaT.SurakkaV., & VanhalaT. (2006). Real-time estimation of emotional experiences from facial expressions. Interacting with Computers, 18(2), 208–226.
93.
PekrunR. (2005). Progress and open problems in educational emotion research. Learning and Instruction, 15(5), 497–506.
94.
PereiraD.OliveiraE., & MoreiraN. (2006). Modelling emotional bdi agents. In Workshop on formal approaches to multi-agent systems (FAMAS 2006), Riva del Garda, Italy.
95.
PereiraD.OliveiraE.MoreiraN., & SarmentoL. (2005). Towards an architecture for emotional bdi agents. In Carlos BentoA. C. & DiasG. (Eds.), 12th Portuguese conference on artificial intelligence (EPIA '05) (pp. 40–46). Universida de da Beira Interior, IEEE ISBN 0–7803–9365–1.
96.
PicardR. (1997). Affective computing.Cambridge, MA: MIT Press.
97.
PicardR. (1998). Toward agents that recognize emotion. In Proceedings of IMAGINA (pp. 153–165), Monaco.
PicardR., & KleinJ. (2002). Computers that recognize and respond to user emotion: Theoretical and practical implications. Interacting with Computers, 14(2), 141–169.
PicardR.VyzasE., & HealeyJ. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 23(10), 1175–1191.
102.
PlutchikR. (1980). A general psychoevolutionary theory of emotion. In PlutchikR. & KellermanH. (Eds.), Emotion: Theory, research, and experience: Vol. 1. Theories of emotion (pp. 3–33). New York: Academic Press.
103.
ReevesB., & NassC. (1996). The media equation: How people treat computers, television, and new media like real people and places.Cambridge: Cambridge University Press.
104.
RogersC. R. (1959). A theory of therapy, personality and interpersonal relationships, as developed in the client-centered framework. In KochS. (Ed.), Psychology: A study of science (Vol. 3; pp. 210–211, 184–256). New York: McGraw-Hill.
105.
RozinP., & CohenA. B. (2003). Confusion infusions, suggestives, correctives, and other medicines. Emotion, 3(1), 92–96.
106.
SchwarzN., & BlessH. (1991). Happy and mindless, but sad and smart? The impact of affective states on analytic reasoning. In ForgasJ. P. (Eds.), Emotion and social judgments (pp. 55–720). New York: Pergamon.
107.
SilvermanB. G.JohnsM.O'BrienK.WeaverR., & CornwellJ. (2002). Constructing virtual asymmetric opponents from data and models in the literature: Case of crowd rioting. In Proceedings of the eleventh conference on computer generated forces and behavioural representation (pp. 97–106), Orlando, Florida.
108.
SlovicP., & LichtensteinS. (1971). Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational Behaviour and Human Performance, 6, 649–744.
109.
SmithE.BartlettM. S., & MovellanJ. (2001). Computer recognition of facial actions: A study of co-articulation effects. In Proceedings of the eighth annual joint symposium on neural computation, La Jolla, California.
110.
SteinN. L., & LevineL. J. (1991). Making sense out of emotion. In KessenW.OrtonyA., & KraikF. (Eds.), Memories, thoughts, and emotions: Essays in honor of George Mandler (pp. 295–322). Hillsdale, NJ: Erlbaum.
111.
SztejnbergA.HurekJ., & AstleitnerH. (2006). FEASP-related emotions of Polish secondary school teachers and students. Journal of Instructional Psychology, 33(1), 63–65.
112.
TickleA. (2000). English and Japanese speaker's emotion vocalizations and recognition: A comparison highlighting vowel quality. In Proceedings of the international speech communication association workshop on speech and emotion (ISCA 2000), Belfast, Northern Ireland.
113.
TriantafillouE.GeorgiadouE., & EconomidesA. A. (2007). The role of user model in CAT (Computer Adaptive Testing): Exploring adaptive variables. Technology, Instruction, Cognition and Learning: An International, Interdisciplinary Journal of Structural Learning, 5(1), 69–89.
114.
Van VugtH. C.HoornJ. F.KonijnE. A., & de Bie DimitriadouA. (2006). Affective affordances: Improving interface character engagement through interaction. International Journal of Human-Computer Studies, 64(9), 874–888.
115.
Van VugtH. C.KonijnE. A.HoornJ. F.KeurI., & EliensA. (2007). Realism is not all! User engagement with task-related interface characters. Interacting with Computers, 19(2), 267–280.
116.
WilsonG. M., & SasseM. A. (2004). From doing to being: Getting closer to the user experience. Interacting with Computers, 16(4), 697–705.
117.
ZamanB., & Shrimpton-SmithT. (2006). The FaceReader: Measuring instant fun of use. In Proceedings of the fourth Nordic conference on human-computer interaction, 457–460, Oslo: ACM Press.
118.
ZhouX., & ConatiC. (2003). Inferring user goals from personality and behaviour in a causal model of user affect. In International Conference on Intelligent User Interfaces (IUI '03), Maimi, FL.
