Every model in this series so far, even the bidirectional LSTM from the last chapter, reads text one step at a time and tries to squeeze everything it has read into a fixed memory. That approach has a ceiling, and the idea that broke through it is attention. Attention is the single most important concept in modern NLP, the foundation of the Transformer and of every large language model. This chapter explains what it is and why it changed the field, building intuition before the architecture itself, which the Transformers chapter of the Neural Networks series covers in full.
Consider machine translation, the task that gave attention its start. The classic approach was an encoder-decoder: one LSTM read the entire source sentence and compressed it into a single fixed-length vector, and a second LSTM expanded that vector into the translation. The flaw is obvious once you see it: no matter how long or complex the sentence, everything has to pass through that one vector. For a long sentence it is like reading a paragraph, then trying to translate it from memory without looking back. Detail gets lost, and the further apart two related words are, the more likely the model loses the connection.
Attention removes the bottleneck with a simple, powerful idea: instead of forcing everything through one vector, let the model look back at all the input words whenever it needs to, and focus on the relevant ones. When the decoder is producing a particular word of the translation, attention lets it look across the whole source sentence and decide which source words matter most right now, then build its decision mostly from those.
Translating an English sentence into French, as the model generates the French word for "cat," attention lets it focus sharply on "cat" in the source while largely ignoring the rest. It computes a set of weights, one per input word, that say how much to focus on each, and uses those weights to form a focused summary. Nothing is lost to a bottleneck, because the original words are always available to look at.
The version that powers modern NLP is self-attention, where a sequence attends to itself. Each word's new representation becomes a weighted blend of every word in the sentence, including itself, with the weights reflecting how relevant each other word is to it. This is exactly what solves the context problem we raised back in the representation chapter. The word "bank" can now build its representation by attending to the surrounding words, so in "river bank" it draws meaning from "river," and in "savings bank" it draws meaning from "savings." The single-vector-per-word limitation of classic embeddings disappears, because every word's vector now depends on its context.
Attention is usually described with three roles, and a search analogy makes them concrete. Each word produces a query (what am I looking for), a key (what do I offer), and a value (the information I carry). To compute attention for a word, you compare its query against every word's key to get relevance scores, turn those scores into weights that sum to one with a softmax, and then take the weighted sum of all the values. Words whose keys match the query well contribute most. Here is that calculation stripped to its essentials:
import numpy as np
def attention(Q, K, V):
scores = Q @ K.T / np.sqrt(K.shape[-1]) # relevance of each key to each query
weights = np.exp(scores) / np.exp(scores).sum(axis=-1, keepdims=True) # softmax
return weights @ V # weighted blend of values
# Q, K, V are matrices: one row per word, derived from the word embeddings.That scaled dot-product is the heart of the Transformer. The division by the square root of the key size is just a scaling trick that keeps the numbers stable, the same numerical-stability concern we met with softmax earlier.
One set of attention weights captures one kind of relationship. Real language has many at once: grammatical links, references like which noun a pronoun points to, topical connections. Multi-head attention runs several attention computations in parallel, each free to focus on a different kind of relationship, then combines them. It is like reading the sentence several times looking for something different each pass.
Attention has two decisive advantages over the recurrent models from earlier. First, it connects any two words directly, no matter how far apart, in a single step, where an LSTM had to carry information across every intervening word and often lost it. Second, because every word's attention can be computed at the same time rather than one step after another, it runs in parallel and trains far faster on modern hardware. Removing the sequential bottleneck is what allowed models to be trained on enormous datasets, which is precisely what made today's large language models possible.
Stack self-attention into a deep architecture, train it on enormous amounts of text, and you get the Transformer, and from it the large language models that now dominate NLP. The next chapter is about exactly those models: BERT, GPT, how they are trained, and how you use them.
Sign in to join the discussion and post comments.
Sign in