Keep a Whoosh index in sync

The first version of a search feature usually rebuilds the whole index on every deploy. That's fine for a few hundred documents, but it gets slow and wasteful once you have thousands — and it means your search is stale between rebuilds. The fix is incremental indexing: only touch the documents that actually changed. Whoosh has everything you need for this built in, with no external service.

pip install whoosh3

1. The one field that makes it possible: a unique key

Give every document a field marked unique=True — a file path, a database primary key, a URL. That key is how Whoosh knows two documents are "the same thing" across runs. Store a mtime (or a version number, or a content hash) so you can tell when a document has changed.

from whoosh.fields import Schema, ID, TEXT, STORED

schema = Schema(
    path=ID(unique=True, stored=True),   # the stable identity of each doc
    mtime=STORED,                        # cheap change-detection signal
    content=TEXT(stored=True),
)

update_document() relies on that unique=True field: it deletes any existing document with the same key and adds the new one, in a single step. Without a unique field, update_document behaves just like add_document and you'll get duplicates.

2. A reusable sync function

The whole strategy is a three-way diff between the current state of your source (here, a directory of files) and the current contents of the index:

1. Read the source: path → mtime for everything on disk.
2. Read the index: path → mtime for everything indexed, via searcher.all_stored_fields().
3. Anything indexed but no longer on disk → delete_by_term. Anything new → add_document. Anything whose mtime grew → update_document. Unchanged files are skipped entirely.

import os
from whoosh import index

def sync_index(ix, docs_dir):
    """Sync the index to the current state of docs_dir.
    Returns (added, updated, deleted)."""
    # 1. what's on disk now
    on_disk = {}
    for name in os.listdir(docs_dir):
        p = os.path.join(docs_dir, name)
        if os.path.isfile(p):
            on_disk[p] = os.path.getmtime(p)

    # 2. what's in the index now
    indexed = {}
    with ix.searcher() as s:
        for fields in s.all_stored_fields():
            indexed[fields["path"]] = fields["mtime"]

    added = updated = deleted = 0
    writer = ix.writer()
    # 3a. delete docs whose file is gone
    for path in set(indexed) - set(on_disk):
        writer.delete_by_term("path", path)
        deleted += 1
    # 3b. add new / update changed
    for path, mtime in on_disk.items():
        if path not in indexed:
            with open(path) as f:
                writer.add_document(path=path, mtime=mtime, content=f.read())
            added += 1
        elif mtime > indexed[path]:
            with open(path) as f:
                writer.update_document(path=path, mtime=mtime, content=f.read())
            updated += 1
    writer.commit()
    return added, updated, deleted

3. Watch it in action

Create the index, sync a directory with two files, then change one file, add a third, and delete the second — and sync again. Only the changed documents are touched:

ix = index.create_in(idxdir, schema)

print("initial:", sync_index(ix, docs))     # first build

# ...then a.txt is edited, c.txt is created, b.txt is removed...
print("resync :", sync_index(ix, docs))
print("noop   :", sync_index(ix, docs))     # nothing changed

with ix.searcher() as s:
    qp = QueryParser("content", ix.schema)
    for term in ["fox", "dog", "cats", "jumps"]:
        n = len(s.search(qp.parse(term)))
        print(f"  '{term}' -> {n} hit(s)")

Verbatim output:

initial: (2, 0, 0)
resync : (1, 1, 1)
noop   : (0, 0, 0)
  'fox' -> 1 hit(s)
  'dog' -> 0 hit(s)
  'cats' -> 1 hit(s)
  'jumps' -> 1 hit(s)

Read the counts: the first sync added both files; the second added the new c.txt, updated the edited a.txt, and deleted the removed b.txt; the third sync found nothing to do. Search reflects reality instantly — dog (only in the deleted file) returns zero, cats (the new file) returns one, and jumps (added in the edit) returns one.

4. Deleting: delete_by_term vs. delete_by_query

To remove a single known document, delete by its unique key:

writer = ix.writer()
writer.delete_by_term("path", "/docs/old.txt")
writer.commit()

To remove a whole batch matching a query — say every document tagged draft, or everything under a path prefix — use delete_by_query:

from whoosh.query import Term, Prefix

writer = ix.writer()
writer.delete_by_query(Term("status", "draft"))
writer.delete_by_query(Prefix("path", "/docs/archive/"))
writer.commit()

Both mark documents as deleted; the space is reclaimed when segments merge during later commits (or you can call ix.optimize() to force it).

Change detection beyond mtime

Gotchas

• No unique=True field? update_document can't find the old copy, so it just adds a new one and you accumulate duplicates. The unique field is mandatory for incremental updates.
• One writer at a time. Whoosh allows a single writer per index; wrap ix.writer() in your own lock (or use AsyncWriter) if multiple processes might index concurrently.
• Batch the commit. Do all the adds/updates/deletes with one writer and a single commit() — committing per document is dramatically slower. See the fast-indexing guide.
• Deleted space isn't freed immediately. Deletions are logical until a merge; call ix.optimize() occasionally if the index keeps growing.

This same sync loop backs the command-line file search and slots directly into a Flask or FastAPI app — index once at startup, then call sync_index() on a schedule or a file-watcher event. For indexing rows instead of files, see searching a pandas DataFrame.