Improving recall (finding more matches)¶

Overview¶

The most common complaint about a freshly-built search index is not that it returns wrong results — it’s that it returns too few. A user searches for run and misses documents that say “running”; they search for car and miss the ones that only mention “automobile”; they make a typo and get nothing at all. This is the recall problem: relevant documents exist, but the query never reaches them.

You do not need embeddings or a vector database to fix most of this. Whoosh ships several complementary recall levers that work on ordinary inverted indexes and run entirely in pure Python. This guide walks through them, from cheapest to most involved:

1. Stem at index time — collapse word forms at index time.
2. Expand variations at query time — expand word forms at query time.
3. Tolerate typos with fuzzy terms — tolerate typos and spelling variants.
4. Offer “did you mean … ?” — suggest a correction when a query misses.
5. Expand a query from its own results (pseudo-relevance feedback) — expand a query with terms from its own top results.

Each technique below is self-contained and independently useful; you can mix and match. The examples share this tiny index:

from whoosh.fields import Schema, TEXT, ID
from whoosh.analysis import StemmingAnalyzer
from whoosh import index
from whoosh.qparser import QueryParser, FuzzyTermPlugin
from whoosh.query import Variations
import tempfile

schema = Schema(id=ID(stored=True),
                body=TEXT(analyzer=StemmingAnalyzer(), stored=True))
ix = index.create_in(tempfile.mkdtemp(), schema)
w = ix.writer()
docs = [
    ("1", "We are connecting the two networks that were disconnected."),
    ("2", "The connection dropped, so the client reconnected automatically."),
    ("3", "Automobiles and cars share many mechanical parts."),
    ("4", "Kubernetes orchestrates containers across clusters at scale."),
]
for i, t in docs:
    w.add_document(id=i, body=t)
w.commit()

1. Stem at index time¶

A StemmingAnalyzer reduces related word forms to a common root before they hit the index, so a query for one form matches all of them. Because the schema above already uses it, a search for connect finds the documents that only say “connecting”, “connection” and “reconnected”:

with ix.searcher() as s:
    qp = QueryParser("body", ix.schema)
    print([h["id"] for h in s.search(qp.parse("connect"))])
    # ['1', '2']

Stemming is the single highest-leverage recall change you can make, and it is essentially free at query time. See Stemming, variations, and accent folding for the trade-offs (it is aggressive and language-specific) and for how to combine it with a Filter that preserves the original word.

Note

Stemming is a heuristic, not a dictionary lookup. The Porter stemmer maps “running” to runn but “run” to run, so those two do not match under stemming alone. When you need to bridge irregular forms, reach for 2. Expand variations at query time or 3. Tolerate typos with fuzzy terms below.

2. Expand variations at query time¶

If you did not stem at index time — for example on an existing index you don’t want to rebuild — you can expand a term into its morphological variations at query time with whoosh.query.Variations. It behaves like a Term query but matches any indexed word that shares the same root:

with ix.searcher() as s:
    print([h["id"] for h in s.search(Variations("body", "run"))])
    # matches documents containing runner / running / runs

The cost is paid per query instead of once at index time, and you keep the original words in the index (useful for exact-phrase search and highlighting). Use Variations when you can’t re-index; use stemming when you can.

3. Tolerate typos with fuzzy terms¶

Add the FuzzyTermPlugin and users can append ~ to a term to match within a small edit distance. kubernets~1 (one edit away) still finds the document about “Kubernetes”:

with ix.searcher() as s:
    qp = QueryParser("body", ix.schema)
    qp.add_plugin(FuzzyTermPlugin())
    print([h["id"] for h in s.search(qp.parse("kubernets~1"))])
    # ['4']

Keep the edit distance small (~1, at most ~2) — larger values match a lot of unrelated words and slow queries down. Fuzzy matching is best reserved for a fallback pass or for fields with lots of proper nouns and product names.

4. Offer “did you mean … ?”¶

When a query returns nothing, you can suggest a nearby indexed term instead of showing an empty page. A Corrector built from the field’s own vocabulary turns a typo into a real word:

with ix.searcher() as s:
    corr = s.corrector("body")
    print(corr.suggest("automobl", limit=3))   # ['automobil']
    print(corr.suggest("connecton", limit=3))  # ['connect']

(The suggestions are stemmed here because the field uses a stemming analyzer; on a non-stemmed field you’d get the surface forms.) This pairs naturally with fuzzy search: try the query as typed, and if the result set is empty, show the top correction as a one-click “search instead for …” link. See “Did you mean… ?” Correcting errors in user queries for word-list backends and tuning.

5. Expand a query from its own results (pseudo-relevance feedback)¶

Sometimes the vocabulary gap is conceptual, not morphological: the user types car but the best documents say “automobile”. Pseudo-relevance feedback (PRF) closes that gap without any external thesaurus. It assumes the top few results are relevant, extracts their most distinctive terms, and uses those to broaden the search. Results.key_terms does the extraction:

with ix.searcher() as s:
    qp = QueryParser("body", ix.schema)
    results = s.search(qp.parse("car"))
    expansion = [t for t, score in results.key_terms("body", numterms=5)]
    print(expansion)
    # ['automobil', 'car', 'mani', 'mechan', 'part']

Notice that automobil surfaced even though the user never typed it. You can feed these terms back into a broadened Or query to pull in documents the original query missed. The closely related Searcher.more_like does the whole round-trip for you — given a document, it finds others like it — which is handy for a “related results” panel.

PRF works best on medium-to-large corpora where the top results are genuinely on-topic; on a tiny or noisy result set it can drift. Gate it behind a minimum result count, or only expand when the original query returned few hits.

Putting it together¶

These levers stack. A pragmatic recall pipeline looks like:

Index with a StemmingAnalyzer (broad, cheap recall).
Run the query as typed.
If the result set is empty or thin, retry with fuzzy terms and surface a did-you-mean suggestion.
When you have a healthy result set, optionally apply PRF to reach conceptually-related documents.

Tip

Recall and precision trade off against each other — every technique here also lets in some noise. The only reliable way to tune them is to build an evaluation set from real user queries (not sentences copied out of your own documents, which flatters recall) and measure the effect of each lever on that set before shipping it.