Visualization of Poincaré Embeddings
M. Nickel and D. Kiela proposed Poincaré Embeddings for learning hierarchical representations (NIPS’17).
One of their tasks consists in embedding words in the Poincaré ball as to preserve entailment links.
Consider a symbolic dataset of words and directed edges between them, indicating that is a subconcept of . Optimize the following
denotes word embeddings parameters,
denotes observed entailment links ,
is the set of negative parents of .
In order to better understand the training dynamics induced by this model, we decided to visualize it in 2D, for the mammal subtree of the WordNet hierarchy.
Using the Poincaré disk distance for , and optimizing parameters using Riemannian SGD, for embeddings in dimension 2, yields the following dynamics:
In comparison, using the Euclidean distance for and optimizing parameters using SGD, for embeddings in dimension 2, yields the following dynamics:
In both simulations, initialization was done with a burn-in phase, i.e. with small learning rate, as suggested by Nickel and Kiela.