Topic Modelling

What is Topic Modelling?

Topic modelling, in simple terms, is a way to discover the topics that are present in a collection of documents (a.k.a corpus). It’s an unsupervised technique, you never tell it what the topics are. It reads the corpus and infers them from which words co-occur across documents.

Soft clustering, not hard clustering

Topic modelling is often taught as a kind of clustering, and that’s a fair intuition, but it’s actually soft clustering. A hard clusterer (like k-means) would put each document in exactly one bucket. A topic model on the other hand, gives each document a distribution over topics:

$$\text{doc}=0.6\cdot {\text{topic}}_A+0.3\cdot {\text{topic}}_B+0.1\cdot {\text{topic}}_C$$

In other words, a document is not assigned to a single topic, but is instead a “scorecard” of how much it belongs to each topic. This is actually a more realistic model of how humans write, because texts are mostly about multiple themes.

NOTE

This “a document is a mixture of topics” property is what makes it a latent-factor model instead of plain clustering.

First, remove the stopwords

Topic modelling is bag-of-words: it only sees which words co-occur, so the most frequent words dominate. The problem is that the most frequent words in any English corpus are stopwords — “the”, “and”, “of”, “to” — which carry no topic signal. Leave them in and every topic looks the same: a cloud of glue words with no meaning.

"Token" and "vector" aren't quite the LLM ones

Here a token is a whole word (not an LLM’s subword chunk), and a document’s vector is a sparse count over the vocabulary: one slot per word, holding how often it appears. Meaning lives nowhere in that geometry: “car” and “automobile” are as unrelated as “car” and “banana”. That’s the opposite of an LLM embedding, a dense learned vector where synonyms sit close together. Swapping count vectors for embeddings is exactly what BERTopic (below) does.

So preprocessing strips them out before training. In practice that means matching each token against a stopword list and dropping the hits:

from nltk.corpus import stopwords
 
def clean(text: str) -> list[str]:
    stop_en = set(stopwords.words("english"))
    return [w for w in text.lower().split() if w not in stop_en and w.isalpha()]

Two things worth knowing:

• The list isn’t universal. You might need to add domain junk to it. For example, in a legal corpus “court” or “plaintiff” may be so ubiquitous they behave like stopwords.
• CountVectorizer can do it for you. CountVectorizer(stop_words="english") removes them during vectorisation, and filtering by document frequency (max_df=0.9) catches corpus-specific ones automatically. Given that a word in 90% of documents can’t tell any two apart.

How LDA works

Latent Dirichlet Allocation is the classic method. The easiest way to understand it is to imagine writing a document the way LDA thinks it was written:

1. A topic is just a bag of words with weights — a “sports” topic leans heavy on game, team, score; a “finance” topic on market, price, stock.
2. To write a document, you first decide its mix of topics — say 70% sports, 30% finance.
3. Then for each word, you pick a topic according to that mix, and pull a word out of its bag.

Of course nobody actually writes like this. LDA just pretends they do, then runs the story backwards: it sees the finished documents and works out the topic bags and per-document mixes that would most likely have produced them. What you get back is two tables — what words make up each topic, and what topics make up each document.

from gensim import corpora
from gensim.models import LdaModel
 
docs = [["bread", "milk", "butter"], ["stock", "market", "price"]]
dictionary = corpora.Dictionary(docs)
bow = [dictionary.doc2bow(d) for d in docs]
 
lda = LdaModel(bow, num_topics=5, id2word=dictionary, passes=10)
for topic in lda.print_topics():
    print(topic)

WARNING

You pick num_topics up front, and LDA won’t tell you if it’s wrong. Too few and topics blur together; too many and they fragment. It’s the k-means k problem wearing a different hat.

Where it’s used

Organising large document sets (news archives, research papers), tagging support tickets by theme, exploratory analysis of open-ended survey responses, and as a feature-extraction step feeding a downstream classifier.

Why LDA at all, in the LLM era?

Honestly, I’m not sure you’d reach for LDA anymore. Ask an LLM to read a document and name its themes and you’ll usually get more coherent, human-readable topics than LDA’s bag-of-words clouds, without num_topics guessing.

Where LDA (or embedding-based methods like BERTopic) still earns its place IMO:

• You want to explore a corpus of documents without paying per document to an LLM API.
• You want to run privately on your own machine, without sending data to a third party.

Besides that, LLMs are usually better at the job. They can read the whole document, not just the bag of words, and they can use their world knowledge to name topics in a way that makes more sense to us.

NOTE

A pragmatic middle ground: cluster document embeddings (BERTopic), then use an LLM only to name each cluster. You get the cheap, corpus-wide structure of a statistical method and the readable labels of an LLM, without paying per document for the modelling itself.