Recently, growing research has focused on analyzing the emotional impact of films, driven by the potential to enhance audience engagement, improve content recommendation systems, and deepen narrative understanding. Advances in artificial intelligence (AI) have opened new avenues for automated emotion recognition across various domains. This study explores the integration of AI, specifically ChatGPT, with human expertise to analyze complex emotional dynamics in cinema. Using a key scene from Sophie's Choice, we investigated ChatGPT's ability to recognize and interpret emotions through a multimodal approach combining visual, auditory, and textual inputs. Two human judges segmented 56 sequences based on characters’ facial expressions, and AI tools were employed for emotion analysis of facial recognition, dialogues, audio cues, and cinematographic elements, through an approach that simulated a real-world analytical workflow and the output of several libraries. Results highlight ChatGPT's ability in detecting nuanced emotional cues, such as micro-expressions and contextual elements, while acknowledging challenges in low-visibility conditions. This study demonstrates AI's capability to decode emotions and narrative dynamics and highlights the importance of multimodal frameworks in improving AI-driven emotion recognition. Future research should examine fully autonomous AI analyses and integrate subjective viewer experiences to bridge the gap between AI interpretations and human emotional responses.
AlbertM.RussellJ. A. (1976). An Approach to Environmental Psychology. M.I.T. Press.
2.
AlmeidaJ.VilaçaL.TeixeiraI. N., et al. (2021). Emotion identification in movies through facial expression recognition. Applied Sciences, 11(15), 1–19.https://doi.org/10.3390/app11156827
3.
BalconiM.CanavesioY. (2016). Is empathy necessary to comprehend the emotional faces? The empathic effect on attentional mechanisms (eye movements), cortical correlates (N200 event-related potentials) and facial behaviour (electromyography) in face processing. Cognition and Emotion, 30(2), 210–224. https://doi.org/10.1080/02699931.2014.993306
4.
BalconiM.CarreraA. (2011). Cross-modal integration of emotional face and voice in congruous and incongruous pairs: The P2 ERP effect. Journal of Cognitive Psychology, 23(1), 132–139. https://doi.org/10.1080/20445911.2011.473560
5.
BalconiM.LucchiariC. (2008). Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis. International Journal of Psychophysiology, 67(1), 41–46. https://doi.org/10.1016/j.ijpsycho.2007.10.002
6.
BalconiM.MazzaG. (2010). Lateralisation effect in comprehension of emotional facial expression: A comparison between EEG alpha band power and behavioural inhibition (BIS) and activation (BAS) systems. Laterality, 15(3), 361–384. https://doi.org/10.1080/13576500902886056
7.
BalconiM.PozzoliU. (2009). Arousal effect on emotional face comprehension: Frequency band changes in different time intervals. Physiology & Behavior, 97(3-4), 455–462. https://doi.org/10.1016/j.physbeh.2009.03.023
8.
BarraP.MnasriZ.GrecoD. (2023). Multimodal emotion recognition from voice and video signals. In EUROCON 2023 - 20th international conference on smart technologies, proceedings (pp. 169–174). Institute of Electrical and Electronics Engineers Inc.
BartschA. (2012). Emotional gratification in entertainment experience. Why viewers of movies and television series find it rewarding to experience emotions. Media Psychology, 15(3), 267–302. https://doi.org/10.1080/15213269.2012.693811
11.
ChengZ.ChengZ.-Q.HeJ.-Y., et al. (2024). Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) (pp. 1–49). NIPS Paper. http://arxiv.org/abs/2406.11161.
12.
CloreG. L.OrtonyA. (2008). Appraisal theories: How cognition shapes affect into emotion. In LewisM.Haviland-JonesJ. M.BarrettL. F. (Eds.), Handbook of emotions (pp. 628–642). Guilford Press.
13.
Cohen-KalafM.LanirJ.BakP., et al. (2022). Movie emotion map: An interactive tool for exploring movies according to their emotional signature. Multimedia Tools and Applications, 81(1), 14663–14684. https://doi.org/10.1007/s11042-021-10803-5
14.
ConnollyH. L.LefevreC. E.YoungA. W., et al. (2020). Emotion recognition ability: Evidence for a supramodal factor and its links to social cognition. Cognition, 197(1), 104166. https://doi.org/10.1016/j.cognition.2019.104166
15.
EkmanP.FriesenW. V. (1976). Measuring facial movement. Environmental Psychology and Nonverbal Behavior, 1(2), 56–75. https://doi.org/10.1007/BF01115465
16.
EkmanP.KeltnerD. (1997). Universal facial expressions of emotion: An old controversy and new findings. In SegerstråleU.MolnárP. (Eds.), Nonverbal communication: Where nature meets culture (pp. 27–46). Lawrence Erlbaum.
17.
ElfenbeinH. A.AmbadyN. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128(3), 203–235. https://doi.org/10.1037/0033-2909.128.2.203
18.
ElyosephZ.Hadar-ShovalD.AsrafK., et al. (2023). ChatGPT outperforms humans in emotional awareness evaluations. Frontiers in Psychology, 14(1), 1199058. https://doi.org/10.3389/fpsyg.2023.1199058
19.
ElyosephZ.RefouaE.AsrafK., et al. (2024). Capacity of generative AI to interpret human emotions from visual and textual data: Pilot evaluation study. JMIR Mental Health, 11(2), e54369. https://doi.org/10.2196/54369
20.
EpsteinM. H.HarnissM. K.PearsonN., et al. (1999). The behavioral and emotional rating scale: Test-retest and inter-rater reliability. Journal of Child and Family Studies, 8(3), 319–327. https://doi.org/10.1023/A:1022067329751
21.
FengS.SunG.LubisN., et al. (2024). Affect recognition in conversations using large language models. In Proceedings of the 25th annual meeting ofthe special interest group on discourse and dialogue (pp. 259–273). Association for Computational Linguistics.
22.
Harmon-JonesE.GableP. A. (2018). On the role of asymmetric frontal cortical activity in approach and withdrawal motivation: An updated review of the evidence. Psychophysiology, 55(2), e12879. https://doi.org/10.1111/psyp.12879
23.
KalatehS.Estrada-JimenezL. A.Nikghadam-HojjatiS., et al. (2024). A systematic review on multimodal emotion recognition: Building blocks, current state, applications, and challenges. In IEEE Access (pp. 103976–104019). Institute of Electrical and Electronics Engineers Inc.
24.
Kesler-WestM. L.AndersenA. H.SmithC. D., et al. (2001). Neural substrates of facial emotion processing using fMRI. Cognitive Brain Research, 11(1), 213–226. https://doi.org/10.1016/S0926-6410(00)00073-2
25.
KhareS. K.Blanes-VidalV.NadimiE. S., et al. (2024). Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations. Information Fusion, 102(2), 102019. https://doi.org/10.1016/j.inffus.2023.102019
26.
KoptyraB.NgoA.RadlińskiŁ, et al. (2023). CLARIN-Emo: Training emotion recognition models using human annotation and ChatGPT. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (pp. 365–379). Springer Science and Business Media Deutschland GmbH.
27.
LianZ.SunL.SunH., et al. (2024). GPT-4V with emotion: A zero-shot benchmark for generalized emotion recognition. Information Fusion, 108(1), 102367. https://doi.org/10.1016/j.inffus.2024.102367
MattioliM.CabitzaF. (2024). Not in my face: Challenges and ethical considerations in automatic face emotion recognition technology. Machine Learning and Knowledge Extraction, 6(1), 2201–2231. https://doi.org/10.3390/make6040109
MohantaS. R.VeerK. (2022). Trends and challenges of image analysis in facial emotion recognition: a review. Network Modeling Analysis in Health Informatics and Bioinformatics, 11, 1–14.https://doi.org/10.1007/s13721-022-00376-0
32.
NadeemM.SohailS. S.JavedL., et al. (2024). Vision-Enabled large language and deep learning models for image-based emotion recognition. Cognito Computer, 16(2), 2566–2579. https://doi.org/10.1007/s12559-024-10281-5
33.
PakulaA. J. (1982). Sophie’s Choice. USA, 1982.
34.
ParkC.KimJ. (2024). Exploring Affective Representations in Emotional Narratives: An Exploratory Study Comparing ChatGPT and Human Responses. Cyberpsychology, Behavior, and Social Networking, 27(2), 1–12.https://doi.org/10.1089/cyber.2024.0100
35.
PiçarraN.ReisE.ChambelT., et al. (2022). Searching, navigating, and recommending movies through emotions: A scoping review. Human Behavior and Emerging Technologies, 2022(1), 7831013. https://doi.org/10.1155/2022/7831013
36.
PollakS. D.SinhaP. (2002). Effects of early experience on children’s recognition of facial displays of emotion. Developmental Psychology, 38(3), 784–791. https://doi.org/10.1037/0012-1649.38.5.784
37.
RottenbergJ.RayR. D.GrossJ. J. (2007). Emotion elicitation using films. In Handbook of emotion elicitation and assessment (pp. 9–28). Oxford University Press.
38.
RussellJ. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(1), 1161–1178. https://doi.org/10.1037/h0077714
39.
SayetteM. A.CohnJ. F.WertzJ. M., et al. (2001). A psychometric evaluation of the facial action coding system for assessing spontaneous expression. Journal of Nonverbal Behavior, 25(5), 167–185. https://doi.org/10.1023/A:1010671109788
40.
SchererK. R. (2001). Appraisal processes in emotion: Theory, methods, research. Oxford University Press.
TianL.MuszynskiM.LaiC., et al. (2017). Recognizing induced emotions of movie audiences: Are induced and perceived emotions the same? In 2017 Seventh international conference on affective computing and intelligent interaction (ACII) (pp. 28–35). IEEE.
43.
ZengZ.PanticM.RoismanG. I., et al. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(3), 39–58. https://doi.org/10.1109/TPAMI.2008.52
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
