Learn AI Create the Future — with Pranay
Main site

Classical NLP That Still Works

Keyword and keyphrase extraction

Keyword extraction picks out the handful of words and phrases that actually describe what a document is about, using nothing but word co-occurrence patterns inside that one document.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Keyword extraction picks out the handful of words that best describe what a document is actually about.

Think about skimming a long news article without reading every word. You can still say "this is about traffic and a new signal system". You did that by noticing which few words kept anchoring the article, not by reading it end to end.

Keyword extraction teaches a computer to do the same skim. It uses nothing but patterns in how words sit next to each other inside that one document.

Why it exists

Tagging documents by hand does not scale past a few hundred documents. Someone still has to decide what each news article, support ticket or product review is "about", one at a time.

Search engines need keywords to index pages. Recommendation systems need them to compare articles. Dashboards need them to summarise what a pile of customer feedback is actually saying. All of that needs an automatic way to find a document's defining words. No human should have to read each one first.

How it works

One classic, effective approach — the idea behind an algorithm called RAKE — needs no big dataset and no training. It works on a single document alone, using two simple ideas:

Split on boring words. Common connector words like "the", "of" and "in" almost never carry the topic themselves. They are used to cut the text into candidate chunks — the words between them become candidate keyphrases.

Score by how much a word "pulls its weight" in a phrase. A word that keeps showing up inside longer phrases is more central to the topic. A word that only ever appears alone is not.

   Sentence:  "the adaptive traffic signal system reduces average commute time"

   Split on boring words ("the"):

     candidate phrase: "adaptive traffic signal system reduces average commute time"

   Each word in a long candidate phrase like this one gets a boost,
   because it appears as part of a longer, more specific chunk —
   not as a lonely, generic word on its own.

Score every candidate phrase this way, sort by score, and the top few are your extracted keywords.

Where you have already seen it

  • Auto-tagging on blogging platforms. Suggested tags for a post you have written.
  • News aggregator "topics". The short phrase shown under a headline summarising what the story covers.
  • Resume and job-post matching tools. Pulling out skills and role keywords from free-form text.

Remember this

  • Keyword extraction finds a document's defining words using patterns inside that one document, no training data needed.
  • Splitting on connector words, then scoring how central each word is to longer phrases, is a simple, effective recipe.
  • It works on word statistics alone — it has no idea what the words actually mean.

What to learn next

  • TF-IDF — a related but different idea: scoring a word's importance across many documents, rather than one.
  • Named entity recognition — pulling out specific people, places and organisations, rather than general topic words.
  • Sentence-transformers — the modern, meaning-aware upgrade for finding what a document is really about.

Developer — Code and libraries.

A RAKE-style keyword extractor, built from scratch, on a single real paragraph.

Setup

Nothing to install — this uses only the standard library.

bash
python --version    # 3.9 or newer

Keyword extraction from scratch

keyword_extraction.py
import re
from collections import defaultdict

STOPWORDS = {
    "a", "an", "the", "is", "are", "was", "were", "in", "on", "at", "of",
    "for", "to", "and", "or", "but", "with", "this", "that", "it", "as",
    "be", "by", "from", "has", "have", "had", "not", "so", "very", "can",
    "using",
}

def extract_keywords(text, top_n=5):
    # split on punctuation first, so a candidate phrase never crosses a sentence break
    chunks = re.split(r"[.,;:!?()\n]+", text.lower())

    phrases = []
    for chunk in chunks:
        current = []
        for w in re.findall(r"[a-z0-9]+", chunk):
            if w in STOPWORDS:
                if current:
                    phrases.append(current)
                    current = []
            else:
                current.append(w)
        if current:
            phrases.append(current)

    # word score = (how often it co-occurs with other words in a phrase, plus
    # its own frequency) divided by its frequency — rewards words that anchor
    # longer phrases, over words that only ever appear on their own
    freq = defaultdict(int)
    degree = defaultdict(int)
    for phrase in phrases:
        for w in phrase:
            freq[w] += 1
            degree[w] += len(phrase) - 1
    word_score = {w: (degree[w] + freq[w]) / freq[w] for w in freq}

    phrase_scores = []
    for phrase in phrases:
        score = sum(word_score[w] for w in phrase)
        phrase_scores.append((score, " ".join(phrase)))
    phrase_scores.sort(key=lambda x: -x[0])

    seen, ranked = set(), []
    for score, phrase in phrase_scores:
        if phrase not in seen:
            seen.add(phrase)
            ranked.append((score, phrase))
    return ranked[:top_n]

text = """
Bangalore traffic congestion has become one of the biggest problems
for daily commuters. The city transport department is testing an
adaptive traffic signal system that adjusts signal timing using real
time traffic data. Early results from the adaptive traffic signal
system show a reduction in average commute time.
"""

for score, phrase in extract_keywords(text, top_n=6):
    print(f"  {score:5.2f}  {phrase}")
Output
  13.08  adaptive traffic signal system
  10.08  adaptive traffic signal
   9.33  adjusts signal timing
   9.25  bangalore traffic congestion
   9.25  time traffic data
   9.00  city transport department

Line by line

"adaptive traffic signal system" is the top-scoring phrase, at 13.08. Every word in it — "adaptive", "traffic", "signal", "system" — also shows up as part of the shorter phrase "adaptive traffic signal" and inside "time traffic data", so each word's degree score is high. Four high-scoring words summed together produces the top result.

Splitting on punctuation before splitting on stopwords matters. Without it, the algorithm would glue words together across a full stop, producing meaningless phrases that stretch across two unrelated sentences. This is a real bug worth testing for directly — see "try it yourself".

(degree[w] + freq[w]) / freq[w] is the actual RAKE word score formula. A word appearing only ever alone has degree == 0, so its score collapses to 1.0 — the lowest possible score. A word that consistently anchors long phrases scores much higher.

Common mistakes

Forgetting to split on sentence-ending punctuation. Splitting only on stopwords, and not on periods or commas, lets a candidate phrase span across a full stop into an unrelated next sentence, corrupting every keyword built from it. Always split on both.

Using a stopword list that is too short or from the wrong domain. A generic English stopword list will miss domain-specific filler — "please", "kindly", "regards" in support tickets, for instance — and those words will pollute your top keywords. Extend the stopword list for your specific text type.

Expecting keyword extraction to understand synonyms. "Traffic signal system" and "traffic light system" are scored as entirely separate phrases here, with no idea they mean the same thing. This is a frequency-and-co-occurrence method, not a meaning-aware one — see sentence-transformers for an approach that does understand synonyms.

Try it yourself

Delete the line chunks = re.split(r"[.,;:!?()\n]+", text.lower()) and replace it with chunks = [text.lower()] — removing all punctuation-based splitting, keeping only stopword-based splitting. Re-run and compare the top result: it will now merge words across the sentence boundary between "...for daily commuters." and "The city transport...", producing a nonsensical combined phrase. That is the exact bug the punctuation split exists to prevent.

What to learn next

  • TF-IDF — scoring a word's importance across a whole collection of documents, instead of within one alone.
  • Named entity recognition — a more targeted way to pull out specific names, places and organisations.
  • Sentence-transformers — meaning-aware phrase and document representations.

Researcher — Mathematics and papers.

RAKE, precisely

RAKE — Rapid Automatic Keyword Extraction (Rose, Engel, Cramer & Cowley, 2010) — operates entirely within a single document, requiring no corpus statistics and no training data, which is its main practical advantage over TF-IDF-based extraction.

The algorithm:

  1. Split the text into candidate phrases using stopwords and phrase delimiters (punctuation) as boundaries.
  2. Build a word co-occurrence graph over the surviving candidate phrases: an edge between two words exists if they co-occur within the same candidate phrase, weighted by co-occurrence count.
  3. Score each word: score(w) = deg(w) / freq(w), where deg(w) is the word's degree in the co-occurrence graph (including a self-loop, so deg(w) >= freq(w)) and freq(w) is its raw frequency.
  4. Score each candidate phrase as the sum of its member words' scores.

The deg/freq ratio specifically favours words that occur frequently in combination with other words over words that occur frequently in isolation — the intended effect is to surface multi-word technical terms over generic frequent unigrams.

Comparison to graph-centrality methods

TextRank (Mihalcea & Tarau, 2004) takes a different approach: build a graph where words (or sentences, for the summarization variant) are nodes, edges represent co-occurrence within a sliding window, and apply PageRank's random-walk centrality algorithm to rank nodes. Unlike RAKE's local degree/frequency ratio, TextRank scores propagate through the whole graph — a word connected to other highly-ranked words gains rank itself, iteratively, until convergence. This generally produces different, sometimes better, rankings on longer documents at the cost of an iterative computation instead of a single pass.

YAKE (Campos et al., 2020) is a more recent unsupervised method combining five statistical features per candidate term — casing, position in the document, frequency, relatedness to context words, and dispersion across sentences — into a single score via a weighted combination, without any graph structure at all. Reported to outperform both RAKE and TextRank on several benchmark datasets, particularly on short documents where RAKE's phrase-co-occurrence signal is sparse.

Supervised and embedding-based extraction

KeyBERT (Grootendorst, 2020) reframes keyword extraction as a similarity problem: embed the whole document and each candidate n-gram using a sentence embedding model (see sentence-transformers), then rank candidates by cosine similarity to the document embedding. This captures semantic centrality that frequency-based methods structurally cannot — a keyword can be extracted even if it appears only once, provided it is semantically representative of the document as a whole.

Complexity

RAKE is O(n) in document length for candidate-phrase construction, and O(n) again for scoring, since the co-occurrence graph has at most n edges for a document of n tokens. TextRank's PageRank step adds O(k * |E|) for k iterations to convergence over |E| graph edges. KeyBERT's cost is dominated by one forward pass of the embedding model per candidate phrase, which is orders of magnitude more expensive per document but requires no hand-built stopword list or heuristic scoring function.

Key references

  • Mihalcea, R. & Tarau, P. (2004). TextRank: Bringing Order into Texts. EMNLP.
  • Rose, S., Engel, D., Cramer, N. & Cowley, W. (2010). Automatic Keyword Extraction from Individual Documents. In Text Mining: Applications and Theory, Wiley.
  • Campos, R. et al. (2020). YAKE! Keyword Extraction from Single Documents Using Multiple Local Features. Information Sciences 509, 257–289.
  • Grootendorst, M. (2020). KeyBERT: Minimal Keyword Extraction with BERT. Software and accompanying technical writeup.

Current state and open problems

Unsupervised statistical methods (RAKE, YAKE) remain the practical default when speed and zero setup matter more than the last few points of precision — they need no model download, no GPU, and every score they produce can be traced back to a simple, auditable count. Embedding-based methods (KeyBERT and its successors) are typically preferred where semantic accuracy matters more than latency, and are increasingly the default in production content-tagging pipelines now that small embedding models are cheap to run. The open problem common to all of these methods is keyphrase quality evaluation itself: standard benchmarks (Inspec, SemEval-2010 Task 5) rely on human-assigned gold keyphrases that different annotators frequently disagree on, which caps how meaningfully different extraction methods can be distinguished by benchmark score alone.

What to learn next

  • TF-IDF — the corpus-level counterpart to this single-document technique.
  • Sentence-transformers — the embedding models that power KeyBERT-style semantic extraction.
  • Named entity recognition — a related, more targeted extraction task with learned models instead of statistics.