About Me
My research interests are related to large language models (LLMs) and their embedding spaces. Together with computational linguists and NLP researchers, I have worked on different projects that explored which linguistic properties and biases get encoded in contextual word embeddings. My focus lies on designing visual analytics approaches that support the interactive investigation of patterns in these high-dimensional vectors.
Selected Publications
Exploring LLMs through Visualizations
Gender bias in BERT
Visualizations can be used to compare models according to their word embedding spaces. By measuring the similarity between words in the model’s parameter space for specific concepts (e.g., stereotypical words and different gender), we can see associations that the models have learned from the training data.
Unexpected biases in BLOOM-1b3
Visualizations can also be used to compare model outputs for slightly varying prompts. By representing outputs in a common 2D space and using clustering on output trajectories, we visually group similar outputs for exploration. As showcased here, especially models with fewer parameters produce more biased outputs.
Function word embeddings encode context information
In BERT, function word embeddings have a low self-similarity, especially in the upper layers. It means that the embedding vectors change if these words are used in different contexts. Hence, embedding vectors do not capture only the word semantic meaning. In upper layers, the nearest neighbors of function words do not necessary have the same POS tag, but rather the neighbors are content words from the same sentence.
BERT does not 'understand' function word meaning
Although function words are highly contextualized, the model does not necessarily 'understand' their functional meaning in the sentence. In many cases, BERT makes semantically wrong predictions when varying used quantifiers, negations, and other function word classes.