Whoosh performance tuning: why indexing is slow and how to make it fast

Live demo · All guides · GitHub · Docs

TL;DR. If Whoosh indexing feels slow, the culprit is almost always opening a new writer (and committing) for every document. Batch your writes into a single writer and you can see a ~20× speedup with no other changes. From there: raise limitmb, use multisegment=True for bulk loads, optimize read-heavy indexes, and reuse one searcher across queries.

The #1 mistake: a writer per document

Every writer.commit() flushes a segment to disk and does real bookkeeping. If you open the index, add one document, and commit in a loop, you pay that cost thousands of times. This is the single most common reason people conclude "Whoosh is slow."

# ❌ Anti-pattern: open + commit per document
from whoosh.index import open_dir

for doc_id, text in documents:
    ix = open_dir("indexdir")
    w = ix.writer()
    w.add_document(id=doc_id, content=text)
    w.commit()          # a full flush every iteration

# ✅ Batch: one writer, one commit
from whoosh.index import open_dir

ix = open_dir("indexdir")
w = ix.writer()
for doc_id, text in documents:
    w.add_document(id=doc_id, content=text)
w.commit()              # a single flush at the end

Measured on 500 small documents (Python 3, whoosh3 3.15.0):

Approach	Time
Writer + commit per document	4.9 s
Single writer, one commit	0.22 s
≈ 22× faster, same result, same index.

Need to add documents continuously (a long-running app) rather than in one batch? Use a BufferedWriter, which keeps a writer open and auto-commits on a size or time threshold — you get batching without holding a single commit open forever. See below.

Tune the writer for large indexes

The writer() call takes several knobs. For big bulk loads these matter; for a few hundred documents the defaults are fine.

w = ix.writer(
    limitmb=256,        # RAM per indexing pool before it spills to disk
                        #   (default 128). More RAM = fewer, larger segments.
    procs=1,            # subprocesses for parallel indexing (see caveat below)
    multisegment=True,  # each subprocess writes its own segment; skips the
                        #   final merge — best for one-shot bulk loads
)
for doc_id, text in documents:
    w.add_document(id=doc_id, content=text)
w.commit()

limitmb — the biggest lever after batching. Raising it means fewer intermediate flushes. On a 20,000-document corpus, bumping limitmb and enabling multisegment shaved indexing from ~6.8 s to ~5.8 s (~15%) on top of already batching. The win grows with corpus size and available RAM.
multisegment=True — great for a one-time bulk import: it avoids the merge step by leaving each pool's output as its own segment. The trade-off is more segments, which slightly slows search until you optimize (next section).
procs — parallel indexing across subprocesses. It helps on large corpora and multi-core machines, but has per-process overhead and pickling costs, so it can be slower for small jobs. Benchmark with your real data before committing to it.

Optimize read-heavy indexes

Whoosh writes each commit as a new segment; searches read every segment. Many small segments (especially after a multisegment bulk load, or many incremental commits) add search overhead. If your index is written once and read a lot, merge everything into one segment:

w = ix.writer()
w.commit(optimize=True)     # merge all segments into one

Optimizing is I/O-heavy, so do it after a bulk load or on a schedule — not after every write. For write-heavy workloads, let Whoosh's automatic segment merging handle it and skip manual optimization.

Reuse one searcher across queries

Opening a searcher warms caches and reads segment metadata. If you open a fresh searcher for every query, you throw that away each time. Open one and reuse it for a batch of queries (searchers are cheap to keep, but not free to create):

# ❌ New searcher per query
for q in queries:
    with ix.searcher() as s:
        s.search(parser.parse(q), limit=10)

# ✅ Reuse across a batch
with ix.searcher() as s:
    for q in queries:
        s.search(parser.parse(q), limit=10)

On 200 repeated queries this dropped from 0.60 s to 0.43 s (~28% faster). In a long-lived service, keep a searcher and refresh it after writes with searcher = searcher.refresh(), which reuses the parts of the index that didn't change.

Streaming writes with BufferedWriter

For apps that add documents over time (a crawler, a queue consumer), you don't want one giant commit or one commit per document. BufferedWriter keeps a writer open and flushes automatically:

from whoosh.writing import BufferedWriter

# flush every 500 docs (period=None disables the time-based flush)
writer = BufferedWriter(ix, period=None, limit=500)
try:
    for doc_id, text in stream():
        writer.add_document(id=doc_id, content=text)
finally:
    writer.close()      # flush the remainder

You can also set period=60 to flush at least once a minute so new documents become searchable promptly, regardless of volume.

A quick checklist

✅ One writer for a batch, not a writer per document (the big one).
✅ Raise limitmb for large bulk loads.
✅ multisegment=True for one-shot imports; optimize afterwards if reads dominate.
✅ Reuse a searcher across queries; refresh() it after writes.
✅ Use BufferedWriter for continuous/streaming ingestion.
✅ Only store (stored=True) the fields you actually display — smaller index, faster reads.
✅ Benchmark procs with your data before assuming parallelism helps.

Install the maintained release: pip install whoosh3 · Docs: batch indexing and concurrency · Related: Whoosh for RAG & hybrid search, Whoosh vs SQLite FTS5.

All timings above were measured against whoosh3 3.15.0 on a single machine before publishing; exact numbers depend on your hardware, document size, and schema, so treat them as ratios rather than absolutes. "Whoosh" is the pure-Python search library originally by Matt Chaput; this is an actively maintained continuation.