Text Preprocessing: Tokenization, Stemming, and Lemmatization

Text Preprocessing: Tokenization, Stemming, and Lemmatization

In the introduction we split a sentence on spaces and immediately hit trouble: punctuation stuck to words, casing lost, no real structure. Preprocessing is the step that deals with all of that. It takes raw, messy text and turns it into clean, consistent units a model can work with. It is unglamorous, but in classical NLP it often makes a bigger difference to results than the choice of model. This chapter covers the standard preprocessing steps and the two that confuse people most, stemming and lemmatization.

Why Preprocessing Matters

Two reasons. First, consistency. To a computer, "Apple", "apple", and "apple," are three different strings unless you make them the same, and treating them separately wastes information and bloats your vocabulary. Second, signal. Much of a text is filler that carries little meaning for many tasks, and trimming it lets the model focus on the words that matter. The catch, which we will come back to, is that "what matters" depends on the task, so preprocessing is a set of choices, not a fixed recipe.

Normalization: Lowercasing and Cleaning

The first moves are usually the simplest. Lowercasing collapses "The" and "the" into one token. Cleaning removes the noise you do not want: punctuation, extra whitespace, HTML tags left over from scraping, and sometimes numbers. None of this is mandatory, and each is a decision. Lowercasing "US" (the country) turns it into "us" (the pronoun), which is occasionally a problem, and stripping punctuation throws away the difference between "let's eat, grandma" and "let's eat grandma." For most tasks these trade-offs are worth it, but it is worth knowing you are making them.

Tokenization

Tokenization is splitting text into units called tokens, usually words. We saw that splitting on spaces is too crude, because it leaves punctuation attached and mishandles cases like "don't" and "New York." Proper tokenizers handle these sensibly, and they can also split text into sentences, which matters for tasks that work sentence by sentence.

import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize, sent_tokenize

text = "Dr. Smith isn't home. He'll be back at 5 p.m., he said."

print(word_tokenize(text))
# ['Dr.', 'Smith', 'is', "n't", 'home', '.', 'He', "'ll", 'be', 'back',
#  'at', '5', 'p.m.', ',', 'he', 'said', '.']

print(sent_tokenize(text))
# ['Dr. Smith isn't home.', "He'll be back at 5 p.m., he said."]

Notice how the tokenizer split "isn't" into "is" and "n't", kept "Dr." and "p.m." intact, and did not wrongly break the sentence at "Dr." A good tokenizer encodes a lot of small, language-specific rules so you do not have to.

There is a more modern approach called subword tokenization, which breaks rare words into smaller pieces (so "tokenization" might become "token" and "ization"). This is what large language models use, because it keeps the vocabulary manageable while still handling any word. We will return to it when we reach large language models.

Stop Words

Stop words are the very common words like "the", "is", "at", and "and" that appear everywhere and often carry little meaning on their own. For tasks like topic classification, removing them focuses the model on the content words. But this is another task-dependent choice, and a sharp example is sentiment analysis: the word "not" is a stop word in many lists, yet removing it turns "not good" into "good" and flips the meaning. So remove stop words when they are noise, and keep them when they carry signal.

from nltk.corpus import stopwords
nltk.download('stopwords')

stop = set(stopwords.words('english'))
tokens = ['this', 'movie', 'was', 'not', 'a', 'good', 'film']
filtered = [t for t in tokens if t not in stop]
print(filtered)
# ['movie', 'good', 'film']   <- note 'not' was removed, which can be dangerous

Stemming vs Lemmatization

Both of these reduce a word to a base form so that "running", "runs", and "ran" can be treated as the same idea. They differ in how, and in how much they care about being correct.

Stemming chops off endings using simple rules. It is fast and crude, and the result is not always a real word. The popular Porter stemmer turns "studies" into "studi" and "running" into "run". The output is consistent, which is what matters for matching, even when it is not a dictionary word.

Lemmatization reduces a word to its actual dictionary base form, called the lemma, using vocabulary and often the word's part of speech. It correctly turns "studies" into "study", "better" into "good", and "ran" into "run". It is slower and needs more information, but the output is always a real word and is more accurate.

from nltk.stem import PorterStemmer, WordNetLemmatizer
nltk.download('wordnet')

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

for word in ["studies", "running", "better", "leaves"]:
    print(word,
          "-> stem:", stemmer.stem(word),
          "| lemma:", lemmatizer.lemmatize(word, pos='v'))
# studies  -> stem: studi   | lemma: study
# running  -> stem: run     | lemma: run
# better   -> stem: better  | lemma: better
# leaves   -> stem: leav    | lemma: leave

The rule of thumb: use stemming when speed matters and rough matching is fine, such as in a search index, and use lemmatization when accuracy and readable output matter. Lemmatization is the more common choice in modern classical-NLP pipelines.

How Much Preprocessing Do You Actually Need?

This depends heavily on the model. Classical methods like the bag-of-words and TF-IDF representations in the next chapter benefit a lot from aggressive preprocessing, because they treat words as independent symbols, so collapsing variants and dropping noise genuinely helps. Modern Transformer models are the opposite. They use their own subword tokenizers and learn from context, so they generally want the raw text with minimal cleaning, since lowercasing or removing stop words can throw away signal they would have used. A good habit is to match your preprocessing to your model rather than applying the heaviest pipeline by default.

Common Pitfalls

• Removing "not" and other negations before sentiment analysis, which flips meaning.
• Lowercasing when case matters, such as distinguishing names from common words.
• Applying heavy classical preprocessing before feeding text to a Transformer, which expects raw input.
• Forgetting that preprocessing choices must be identical at training time and prediction time.

Key Takeaways

• Preprocessing turns messy raw text into clean, consistent tokens, and in classical NLP it strongly affects results.
• Tokenization splits text into words or sentences and handles tricky cases that naive splitting gets wrong.
• Stop word removal helps some tasks and hurts others, so it is a choice, not a default.
• Stemming is fast and crude; lemmatization is slower, accurate, and produces real words.
• How much to preprocess depends on the model: heavy for classical methods, light for Transformers.

What's Next

Now that we can turn raw text into clean tokens, we hit the central obstacle of NLP: models work with numbers, not words. The next part of the series is all about bridging that gap, starting with the classical ways to turn text into numbers, bag-of-words and TF-IDF, and then the leap to word embeddings.