Sage Journals: Discover world-class research

Abstract

This paper proposes a sentiment analysis framework based on ranking learning. The framework utilizes BERT model pre-trained on large-scale corpora to extract text features and has two sub-networks for different sentiment analysis tasks. The first sub-network of the framework consists of multiple fully connected layers and intermediate rectified linear units. The main purpose of this sub-network is to learn the presence or absence of various emotions using the extracted text information, and the supervision signal comes from the cross entropy loss function. The other sub-network is a ListNet. Its main purpose is to learn a distribution that approximates the real distribution of different emotions using the correlation between them. Afterwards the predicted distribution can be used to sort the importance of emotions. The two sub-networks of the framework are trained together and can contribute to each other to avoid the deviation from a single network. The framework proposed in this paper has been tested on multiple datasets and the results have shown the proposed framework’s potential.

Keywords

Sentiment analysis multi-label classification ranking

Get full access to this article

View all access options for this article.

References

Agarwal

, Xie

, Vovsha

, Rambow

and Passonneau

, Sentiment analysis of twitter data, in Proceedings of the 2011 ACL Workshop on Language in Social Media, 2011, pp. 30–38.

Godbole

, Srinivasaiah

and Skiena

, Large-scale sentiment analysis for news and blogs, in Proceedings of the First International Conference on Weblogs and Social Media, 2007.

Taboada

, Brooke

, Tofiloski

, Voll

K.D.

and Stede

, Lexicon-based methods for sentiment analysis, Computational Linguistics37 (2) (2011), 267–307.

Esuli

and Sebastiani

, SENTIWORDNET: A publicly available lexical resource for opinion mining, in Proceedings of the 5th International Conference on Language Resources and Evaluation, 2006, pp. 417–422.

Baccianella

, Esuli

and Sebastiani

, Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining, in Proceedings of the 7th International Conference on Language Resources and Evaluation, 2010.

Liang

, Liu

, Tan

and Bai

, Sentiment classification based on AS-LDA model,, Procedia Computer Science31 (2014), 511–516.

and Shen

, Research on sentiment analysis of microblogging based on LSA and TF-IDF, in Proceedings of 3rd IEEE International Conference on Computer and Communications, 2017, pp. 2584–2588.

Kim

, Convolutional neural networks for sentence classification, in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014, pp. 1746–1751.

Day

M.-Y.

and Lin

Y.-D.

, Deep learning for sentiment analysis on Google play consumer review, in Proceedings of 2017 IEEE International Conference on Information Reuse and Integration, 2017, pp. 382–388.

10.

Huang

, Peng

, Li

and Lee

, Sentiment and topic analysis on social media: A multi-task multi-label classification approach, in Proceedings of ACM Web Science 2013 Conference, 2013, pp. 172–181.

11.

Yang

, Zhou

and He

, An interpretable neural network with topical information for relevant emotion ranking, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 3423–3432.

12.

Liu

S. M.

and Chen

, A multi-label classification based approach for sentiment classification, Expert Systems with Applications42 (3) (2015), 1083–1093.

13.

Howard

and Ruder

, Universal language model fine-tuning for text classification, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018, pp. 328–339.

14.

Zhou

, Yang

and He

, Relevant emotion ranking from text constrained with emotion relationships, in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018, pp. 561–571.

15.

, A short introduction to learning to rank, IEICE Transactions on Information Systems94-D (10) (2011), 1854–1862.

16.

Devlin

, Chang

, Lee

and Toutanova

, BERT: Pre-training of deep bidirectional transformers for language understanding, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. 4171–4186.

17.

Cao

, Qin

, Liu

, Tsai

and Li

, Learning to rank: From pairwise approach to listwise approach, in Proceedings of the 24th International Conference on Machine Learning, 2007, pp. 129–136.

18.

Das

S. R.

and Chen

M. Y.

, Yahoo! for amazon: Extracting market sentiment from stock message boards, in Proceedings of the Asia Pacific Finance Association Annual Conference, 2001.

19.

Huettner

and Subasic

, Fuzzy typing for document management, ACL 2000 Companion Volume: Tutorial Abstracts and Demonstration Notes, 2000, pp. 26–27.

20.

and Liu

, Mining and summarizing customer reviews, in Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2004, pp. 168–177.

21.

Schwenk

, Barrault

, Conneau

and LeCun

, Very deep convolutional networks for text classification, in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2017, pp. 1107–1116.

22.

Rong

, Peng

, Ouyang

, Li

and Xiong

, Structural information aware deep semi-supervised recurrent neural network for sentiment analysis, Frontiers of Computer Science9 (2) (2015), 171–184.

23.

Tang

, Qin

, Feng

and Liu

, Effective LSTMs for target-dependent sentiment classification, in Proceedings of the 26th International Conference on Computational Linguistics, 2016, pp. 3298–3307.

24.

Tang

, Qin

and Liu

, Document modeling with gated recurrent neural network for sentiment classification, in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 1422–1432.

25.

Rehman

A.U.

, Malik

A.K.

, Raza

and Ali

, A hybrid CNN-LSTM model for improving accuracy of movie reviews sentiment analysis, Multimedia Tools and Applications78 (18) (2019), 26597–26613.

26.

Gan

, Wang

, Zhang

and Wang

, Sparse attention based separable dilated convolutional neural network for target entities sentiment analysis, Knowledge-Based Systems, 188 (2020).

27.

, Zha

Z.-J.

, Wang

and Chua

T.-S.

, Aspect ranking: Identifying important product aspects from online consumer reviews, in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011, pp. 1496–1505.

28.

Dohaiha

H. H.

, Prasad

, Maag

and Alsadoon

, Deep learning for aspect-based sentiment analysis: A comparative review, Expert Systems With Applications118 (15) (2019), 272–299.

29.

Ruder

, Ghaffari

and Breslin

J. G.

, INSIGHT-1 at semeval-2016 task 5: Deep learning for multilingual aspect-based sentiment analysis, in Proceedings of the 10th International Workshop on Semantic Evaluation, 2016, pp. 330–336.

30.

, Gu

and Wu

, Cascaded convolutional neural networks for aspect-based opinion summary, Neural Processing Letters46 (2) (2017), pp. 581–594.

31.

, Gu

, Sun

and Gu

, Aspect-based opinion summarization with convolutional neural networks, in Proceedings of 2016 International Joint Conference on Neural Networks, 2016, pp. 3157–3163.

32.

Vaswani

, Shazeer

, Parmar

, Uszkoreit

, Jones

, Gomez

A. N.

, Kaiser

and Polosukhin

, Attention is all you need, in Proceedings of 2017 Annual Conference on Neural Information Processing Systems, 2017, pp. 6000–6010.

33.

L.J.

, Kiros

and Hinton

G. E.

, Layer normalization, CoRR, abs/1607.06450, 2016.

34.

Burges

C.J.C.

, Shaked

, Renshaw

, Lazier

, Deeds

, Hamilton

and Hullender

G.N.

, Learning to rank using gradient descent, in Proceedings of the 22nd International Conference on Machine Learning, 2005, pp. 89–96.

35.

, Wang

, Chen

and Li

, Empirical evaluation of rectified activations in convolutional network, CoRR, abs/1505.00853, 2015.

36.

Quan

and Ren

, Construction of a blog emotion corpus for chinese emotional expression analysis, in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 2009, pp. 1446–1454.

37.

Hamming

R. W.

, Error detecting and error correcting codes, The Bell System Technical Journal29 (2) (1950), 147–160.

38.

Järvelin

and Kekäläinen

, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems20 (4) (2002), 422–446.

39.

Kingma

D. P.

and Ba

, Adam:Amethod for stochastic optimization, in Proceedings of 3rd International Conference on Learning Representations, 2015.

40.

Goyal

, Dollár

, Girshick

R. B.

, Noordhuis

, Wesolowski

, Kyrola

, Tulloch

, Jia

and He

, Accurate, large minibatch SGD: training ImageNet in 1 hour, CoRR, abs/1706.02677, 2017.

41.

Felbo

, Mislove

, Søgaard

, Rahwan

and Lehmann

, Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm, in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 1615–1625.

42.

Lin

, Su

, Yang

, Ma

and Sun

, Semantic-unit-based dilated convolution for multi-label text classification, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 4554–4564.