Whoosh analyzer recipes

Most "why doesn't my search match?" problems in Whoosh are really analyzer problems: the query term and the indexed term were tokenized differently, so they never line up. The fix is almost always to pick (or compose) the right analyzer for a field. This page is a practical catalogue. If you want the conceptual background, the docs cover the analysis pipeline in depth; here we go straight to working code.

pip install whoosh3

The one habit that saves hours: inspect your tokens

An analyzer is a callable that yields Token objects. You never have to guess what it does — just call it and read the .text of each token:

from whoosh.analysis import StandardAnalyzer

ana = StandardAnalyzer()
print([t.text for t in ana("The Quick, Brown FOXES jumped!")])

['quick', 'brown', 'foxes', 'jumped']

Notice three things the default StandardAnalyzer did: it lowercased everything, dropped punctuation, and removed the stop word the. If you index a field with this analyzer, a search for "The" or "FOX" behaves accordingly. Seeing the token list first is the fastest way to understand — and debug — any field.

Want to keep short words and stop words (useful for code, product names, or short fields)? Turn them off:

ana = StandardAnalyzer(stoplist=None, minsize=1)
print([t.text for t in ana("The Quick, Brown FOXES jumped!")])

['the', 'quick', 'brown', 'foxes', 'jumped']

Recipe: match word variants (stemming)

Stemming reduces words to a root so running, runner, and ran can match each other. Use StemmingAnalyzer:

from whoosh.analysis import StemmingAnalyzer

ana = StemmingAnalyzer()
print([t.text for t in ana("The runners were running and ran")])

['runner', 'were', 'runn', 'ran']

The Porter stemmer is a heuristic, not a dictionary — running becomes runn and ran stays ran, so irregular verbs won't always collapse together. That's expected and fine for most search. The key point is you must use the same analyzer on the field and the query (Whoosh does this automatically when the query parser knows the schema), so both sides stem identically. This is also the analyzer to reach for when migrating from SQLite FTS5's porter tokenizer.

Recipe: accent-insensitive search (café → cafe)

A classic requirement: a user types resume and should find résumé. Compose your own analyzer with a CharsetFilter using Whoosh's built-in accent_map:

from whoosh.analysis import RegexTokenizer, LowercaseFilter, CharsetFilter
from whoosh.support.charset import accent_map

folding = RegexTokenizer() | LowercaseFilter() | CharsetFilter(accent_map)
print([t.text for t in folding("Café Montréal naïve JALAPEÑO")])

['cafe', 'montreal', 'naive', 'jalapeno']

The | operator chains a tokenizer and filters into a pipeline — this is how you build any custom analyzer in Whoosh. To use it, attach it to a field and index normally; searches then match across accents in both directions:

from whoosh.fields import Schema, ID, TEXT
from whoosh.support.charset import accent_map
from whoosh.analysis import RegexTokenizer, LowercaseFilter, CharsetFilter
from whoosh import index
from whoosh.qparser import QueryParser
import tempfile

folding = RegexTokenizer() | LowercaseFilter() | CharsetFilter(accent_map)
schema = Schema(id=ID(stored=True), body=TEXT(analyzer=folding, stored=True))

ix = index.create_in(tempfile.mkdtemp(), schema)
w = ix.writer()
w.add_document(id="1", body="I love a good Café con leche")
w.commit()

with ix.searcher() as s:
    q = QueryParser("body", ix.schema).parse("cafe")   # no accent typed
    print([hit["id"] for hit in s.search(q)])

['1']

A plain cafe query matched the stored Café, because both were folded to the same token at index and query time.

Recipe: search per-language (French, German, …)

LanguageAnalyzer bundles a language-appropriate stop list and stemmer:

from whoosh.analysis import LanguageAnalyzer

ana = LanguageAnalyzer("fr")
print([t.text for t in ana("Les chats mangent les souris rapidement")])

['le', 'chat', 'mangent', 'le', 'sour', 'rapid']

Pass a two-letter language code ("fr", "de", "es", …). Whoosh ships a pure-Python stemmer, and will use the faster PyStemmer C extension automatically if it happens to be installed — but it is never required, so the no-compiler promise still holds.

Recipe: split camelCase, hyphens, and part numbers

Identifiers like getUserName, WiFi-router, or Model-500 should often be searchable by their pieces. IntraWordFilter splits on case transitions, letter/number boundaries, and internal punctuation:

from whoosh.analysis import RegexTokenizer, IntraWordFilter, LowercaseFilter

ana = RegexTokenizer(r"\S+") | IntraWordFilter() | LowercaseFilter()
print([t.text for t in ana("WiFi-router Model-500 getUserName")])

['wi', 'fi', 'router', 'model', '500', 'get', 'user', 'name']

Now a search for router, 500, or user will hit those documents. This is ideal for code search, SKUs, and log lines. (Note the tokenizer here is RegexTokenizer(r"\S+") so hyphenated runs reach the filter as a single chunk before it splits them.)

Recipe: index a whole value as one token (tags, categories, slugs)

A very common mistake is to reach for KeywordAnalyzer expecting it to keep a multi-word value intact. It doesn't — it splits on whitespace (or commas):

from whoosh.analysis import KeywordAnalyzer

print([t.text for t in KeywordAnalyzer(lowercase=True)("New York City")])

['new', 'york', 'city']

If you want the entire value to become a single exact token — for a tag, category, slug, or facet — use IDAnalyzer (which is what the ID field type uses under the hood):

from whoosh.analysis import IDAnalyzer

print([t.text for t in IDAnalyzer(lowercase=True)("Machine Learning")])

['machine learning']

One token, lowercased for case-insensitive exact match. Use this for fields you filter or facet on rather than do free-text search over.

Recipe: substring / "search-as-you-type" with n-grams

Whoosh's query language supports wildcards, but for fast substring matching (e.g. matching arch inside search) an n-gram analyzer indexes every fixed-length slice:

from whoosh.analysis import NgramAnalyzer

print([t.text for t in NgramAnalyzer(3)("search")])

['sea', 'ear', 'arc', 'rch']

Index a field with NgramAnalyzer and a query for any 3+ character fragment matches. It trades index size for speed and flexibility, so reserve it for fields where substring/autocomplete matters. For prefix-only autocomplete (the more common case), the dedicated prefix-search recipe is lighter weight.

Cheat sheet

Whichever you pick, keep the habit from the top of this page: print the token list first. If the tokens on the page match the tokens in your query, your search will work.

Related recipes: spelling / "did you mean?", autocomplete & prefix search, and fast indexing & performance tuning.