We can now clean and tokenize text. But there is a hard wall between clean tokens and any model: machine learning algorithms work with numbers, and words are not numbers. This chapter is about the bridge. It covers the classical ways to turn text into numeric features, bag-of-words and TF-IDF, then the breakthrough that made modern NLP possible, word embeddings. Understanding all three matters, because the classical methods are still everywhere and the leap from them to embeddings is the single most important idea in the field.
Suppose you have three short reviews and you want a model to tell good from bad. The model needs each review as a vector of numbers. The question is how to produce that vector in a way that captures something useful about the text. Every method below is an answer to that question, and each is better than the last at one thing: capturing meaning.
The simplest approach is to build a vocabulary of every word across all your documents, then represent each document by how many times each vocabulary word appears in it. Order is thrown away, which is why it is called a "bag" of words. The result is a document-term matrix: one row per document, one column per vocabulary word, counts in the cells.
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
"the movie was good",
"the movie was bad",
"the food was good and the service was good",
]
vec = CountVectorizer()
X = vec.fit_transform(corpus)
print(vec.get_feature_names_out())
# ['and' 'bad' 'food' 'good' 'movie' 'service' 'the' 'was']
print(X.toarray())
# [[0 0 0 1 1 0 1 1]
# [0 1 0 0 1 0 1 1]
# [1 0 1 2 0 1 2 2]]Each review is now a vector the model can use. This is simple, fast, and surprisingly effective for many classification tasks. But it has real weaknesses. The vectors are very sparse, almost all zeros, because any one document uses a tiny slice of the full vocabulary. It ignores word order entirely, so "good not bad" and "bad not good" look identical. And it treats every word as unrelated: "good" and "great" are as different to it as "good" and "elephant."
You can recover some word order by counting short sequences instead of single words. A bigram model counts pairs like "not good" and "very bad" as their own features, which lets the representation capture phrases. This helps, but it makes the vocabulary explode, so it is a partial fix rather than a real solution.
Bag-of-words treats a common word like "the" as just as important as a rare, telling word like "refund." TF-IDF fixes this by weighting each word by how informative it is. It multiplies two quantities: term frequency, how often the word appears in this document, and inverse document frequency, which is high for words that appear in few documents and low for words that appear everywhere. The effect is that words common across the whole collection get pushed down, and words distinctive to a particular document get pushed up.
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.fit_transform(corpus)
# "the" and "was" appear everywhere, so they get LOW weights.
# "food", "service", "bad" are distinctive, so they get HIGHER weights.TF-IDF is still the strong, sensible default for many text classification and search tasks, and it often beats far more complex methods on small datasets. But it shares the deeper limitation of bag-of-words: the vectors are still sparse, still ignore most word order, and still have no idea that "good" and "great" mean similar things. Each word is just a column, with no relationship to any other.
Word embeddings are the idea that changed everything. Instead of representing a word as a column in a giant sparse matrix, you represent it as a short, dense vector of real numbers, perhaps 100 or 300 of them, learned so that words used in similar contexts end up with similar vectors. The guiding principle, often quoted, is that you shall know a word by the company it keeps. A model reads enormous amounts of text and learns that "good" and "great" appear in similar places, so it places their vectors close together.
The remarkable result is that these vectors capture meaning in a way you can measure and even do arithmetic with. Similar words sit near each other, and relationships show up as consistent directions in the space. The famous example is that the vector for "king" minus "man" plus "woman" lands very close to "queen." The geometry of the space encodes real semantic structure.
Three methods are worth knowing. Word2Vec learns embeddings by training a small neural network to predict a word from its neighbours, or the neighbours from the word. GloVe learns them from global word co-occurrence statistics across the whole corpus. FastText extends Word2Vec by building word vectors out of subword pieces, which means it can produce a sensible vector even for a word it never saw in training, and it handles misspellings and rare words well.
from gensim.models import Word2Vec
# Each document is a list of tokens
sentences = [
["the", "movie", "was", "good"],
["the", "film", "was", "great"],
["the", "food", "was", "bad"],
]
model = Word2Vec(sentences, vector_size=50, window=3, min_count=1, seed=1)
print(model.wv["movie"].shape) # (50,) a dense 50-dim vector
print(model.wv.most_similar("movie")) # words with nearby vectorsIn practice you rarely train embeddings from scratch on a small corpus like this. You download vectors pretrained on billions of words, the same idea as the transfer learning covered in the Neural Networks series, and use those.
Classic word embeddings have one big blind spot: each word gets exactly one vector, no matter the context. The word "bank" has a single embedding whether you mean a river bank or a savings bank. That is clearly wrong, and fixing it is exactly what modern contextual models do. Transformer models produce a different vector for a word depending on the sentence around it, which is why they are so much more powerful. Those contextual embeddings are also what power semantic search and retrieval systems, a topic the embeddings chapter of the RAG field manual goes into for real applications. We build toward all of this in the chapters on attention and large language models.
Start with TF-IDF for classification and search on small or medium datasets, because it is fast, strong, and easy to interpret. Reach for pretrained word embeddings when you want some sense of meaning and similarity and have a bit more data. Use contextual embeddings from Transformers when you need the best quality and can afford the extra weight, which is most serious modern NLP. The progression from counts to TF-IDF to embeddings to contextual embeddings is, in a sentence, the history of the field.
With ways to represent text in hand, we can start doing real work. The next part of the series covers the core NLP tasks, beginning with named entity recognition: pulling the people, places, and organisations out of raw text.
Sign in to join the discussion and post comments.
Sign in