Sage Journals: Discover world-class research

Abstract

Word embedding, a high-dimensional (HD) numerical representation of words generated by machine learning models, has been used for different natural language processing tasks, for example, translation between two languages. Recently, there has been an increasing trend of transforming the HD embeddings into a latent space (e.g. via autoencoders) for further tasks, exploiting various merits the latent representations could bring. To preserve the embeddings’ quality, these works often map the embeddings into an even higher-dimensional latent space, making the already complicated embeddings even less interpretable and consuming more storage space. In this work, we borrow the idea of $β$ VAE to regularize the HD latent space. Our regularization implicitly condenses information from the HD latent space into a much lower-dimensional space, thus compressing the embeddings. We also show that each dimension of our regularized latent space is more semantically salient, and validate our assertion by interactively probing the encoding-level of user-proposed semantics in the dimensions. To the end, we design a visual analytics system to monitor the regularization process, explore the HD latent space, and interpret latent dimensions’ semantics. We validate the effectiveness of our embedding regularization and interpretation approach through both quantitative and qualitative evaluations.

Keywords

High-dimensional data visualization visual analytics neural networks word embedding

Get full access to this article

View all access options for this article.

References

Liu

Bremer

Thiagarajan

, et al. Visual exploration of semantic relationships in neural word embeddings. IEEE Trans Vis Comput Graph 2018; 24(1): 553–562.

Njotoprawiro

Haleem

, et al. Embeddingvis: a visual analytics approach to comparative network embedding inspection. In: 2018 IEEE VAST, pp.48–59. New York: IEEE.

Shen

Ritter

, et al. Visual exploration of neural document embedding in information retrieval: semantics and feature selection. IEEE Trans Vis Comput Graph 2019; 25(6): 2181–2192.

Mohiuddin

Bari

Joty

. Lnmap: Departures from isomorphic assumption in bilingual lexicon induction through non-linear mapping in latent space. arXiv preprint, arXiv:2004.13889, 2020.

Voigt

Von dem Bussche

. The EU general data protection regulation (GDPR). A practical guide. 1st ed. Cham: Springer International Publishing, 2017, Vol. 10, pp.3152676.

Kingma

Welling

. Auto-encoding variational bayes. arXiv preprint, arXiv:13126114.2013.

Higgins

Matthey

Pal

, et al. β-VAE: learning basic visual concepts with a constrained variational framework. In: 5th International conference on learning representations, ICLR2017.

Higgins

Sonnerat

Matthey

, et al. Scan: learning hierarchical compositional visual concepts. arXiv preprint, arXiv:170703389, 2017.

Wang

Zhang

Yang

. Scanviz: Interpreting the symbol-concept association captured by deep neural networks through visual analytics. In: IEEE Pacific visualization symposium, pp.51–60. New York: IEEE.

10.

Heinrich

Weiskopf

. State of the art of parallel coordinates. In: Eurographics (State of the Art Reports), pp.95–116.

11.

Elmqvist

Dragicevic

Fekete

. Rolling the dice: multidimensional visual exploration using scatterplot matrix navigation. IEEE Trans Vis Comput Graph 2008; 14(6): 1539–1148.

12.

Hoffman

Grinstein

Marx

, et al. DNA visual and analytic data mining. In: Proceedings. Visualization’97 (Cat. No. 97CB36155), pp.437–441. New York: IEEE.

13.

Kandogan

. Star coordinates: A multi-dimensional visualization technique with uniform treatment of dimensions. In: Proceedings of the IEEE information visualization symposium, Vol. 650, p.22. Citeseer.

14.

Van der Maaten

Hinton

. Visualizing data using t-SNE. J Mach Learn Res 2008; 9(11): 2579–2605.

15.

McInnes

Healy

Melville

. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint, arXiv:180203426, 2018.

16.

Choo

Liu

. Visual analytics for explainable deep learning. IEEE Comput Graph Appl 2018; 38(4): 84–92.

17.

Yuan

Chen

Yang

, et al. A survey of visual analytics techniques for machine learning. Comput Vis Media 2021; 7(1): 3–36.

18.

Yang

Liu

, et al. Diagnosing concept drift with visual analytics. In: 2020 IEEE conference on visual analytics science and technology (VAST), pp.12–23. New York: IEEE.

19.

Liu

Shi

, et al. Towards better analysis of deep convolutional Neural Networks. IEEE Trans Vis Comput Graph 2017; 23(1): 91–100.

20.

Ren

Amershi

Lee

, et al. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE Trans Vis Comput Graph 2017; 23(1): 61–70.

21.

Rathore

Dev

Phillips

, et al. Verb: Visualizing and interpreting bias mitigation techniques for word representations. arXiv preprint, arXiv:210402797, 2021.

22.

El-Assady

Kehlbeck

Collins

, et al. Semantic concept spaces: guided topic model refinement using word-embedding projections. IEEE Trans Vis Comput Graph 2020; 26(1): 1001–1011.

23.

Park

Kim

Lee

, et al. Conceptvector: text visual analytics via interactive lexicon building using word embedding. IEEE Trans Vis Comput Graph 2018; 24(1): 361–370.

24.

Liu

Jun

, et al. Latent space cartography: visual analysis of vector space embeddings. Comput Graph Forum 2019; 38: 67–78.

25.

Wang

Gou

Zhang

, et al. Deepvid: Deep visual interpretation and diagnosis for image classifiers via knowledge distillation. IEEE Trans Vis Comput Graph 2019; 25(6): 2168–2180.

26.

Gou

Zou

, et al. Vatld: a visual analytics system to assess, understand and improve traffic light detection. IEEE Trans Vis Comput Graph 2021; 27(2): 261–271.

27.

Burgess

Higgins

Pal

, et al. Understanding disentangling in β-VAE. arXiv preprint, arXiv:1804.03599, 2018.

28.

Camacho-Collados

Pilehvar

Collier

, et al. Semeval-2017 task 2: Multilingual and cross-lingual semantic word similarity. In: Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pp.15–26. Vancouver: Association for Computational Linguistics. http://www.aclweb.org/anthology/S17-2002

29.

Mikolov

Chen

Corrado

, et al. Efficient estimation of word representations in vector space. arXiv preprint, arXiv:13013781, 2013.

30.

Conneau

Lample

Ranzato

, et al. Word translation without parallel data. arXiv preprint, arXiv:171004087, 2017.

31.

Mikolov

Sutskever

. Exploiting similarities among languages for machine translation. arXiv preprint, arXiv:1309.4168, 2013.

32.

Ruder

Vulić

Søgaard

. A survey of cross-lingual word embedding models. J Artif Intell Res 2019; 65: 569–631.

33.

Bojanowski

Grave

Joulin

, et al. Enriching word vectors with subword information. Trans Assoc Comput Linguist 2017; 5: 135–146.

34.

Inselberg

Dimsdale

. Parallel coordinates: a tool for visualizing multi-dimensional geometry. In: Proceedings of the First IEEE conference on visualization, pp.361–378. New York: IEEE.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.16 MB

59.19 MB

0.00 MB