Search a folder of PDFs & Markdown with Whoosh

You have a directory of docs — meeting notes, Markdown wikis, exported PDFs — and you want to search across all of them by keyword, with highlighted snippets, in milliseconds, on your own machine. No Elasticsearch cluster, no external service. Whoosh is a good fit: it's pure Python, embeds directly in your app, and the index is just files in a folder you can ship anywhere.

1. A schema that tracks files

Two design choices make the sync logic easy. Make path a unique field so re-indexing the same file replaces its entry instead of duplicating it, and store the file's mtime so a later run can tell what actually changed.

from whoosh.fields import Schema, ID, TEXT, STORED

SCHEMA = Schema(
    path=ID(unique=True, stored=True),   # unique -> update_document replaces
    mtime=STORED,                        # modification time, for change detection
    title=TEXT(stored=True),
    content=TEXT(stored=True),           # stored so we can highlight snippets
)

2. Turn each file into text

Whoosh indexes text, so you need one small function that reads a file and returns plain text. Keep it pluggable per extension — text files are trivial, Markdown just needs its markup stripped, and PDFs use a library like pypdf to pull text out of each page.

import os, re

def extract_text(path):
    ext = os.path.splitext(path)[1].lower()
    if ext == ".pdf":
        from pypdf import PdfReader                     # pip install pypdf
        reader = PdfReader(path)
        return "\n".join(page.extract_text() or "" for page in reader.pages)
    if ext in (".md", ".markdown"):
        raw = open(path, encoding="utf-8", errors="ignore").read()
        return re.sub(r"[#*`>_\-]", " ", raw)            # strip common Markdown marks
    # .txt and anything else: read as text
    return open(path, encoding="utf-8", errors="ignore").read()

3. Open (or create) the index

from whoosh import index

def get_index(index_dir):
    if index.exists_in(index_dir):
        return index.open_dir(index_dir)
    os.makedirs(index_dir, exist_ok=True)
    return index.create_in(index_dir, SCHEMA)

4. Incremental sync: add, update, delete

This is the heart of it. Read the paths and mtimes already in the index, walk the folder, and only touch files that are new or changed. Because path is unique, update_document handles both insert and replace. Finally, drop any indexed file that no longer exists on disk.

def index_folder(docs_dir, index_dir, exts=(".txt", ".md", ".markdown", ".pdf")):
    ix = get_index(index_dir)

    # What's already indexed, and when it was last modified?
    indexed = {}
    with ix.searcher() as s:
        for fields in s.all_stored_fields():
            indexed[fields["path"]] = fields["mtime"]

    on_disk = set()
    added = updated = removed = 0
    writer = ix.writer()

    for root, _, files in os.walk(docs_dir):
        for name in files:
            if not name.lower().endswith(exts):
                continue
            path = os.path.join(root, name)
            on_disk.add(path)
            mtime = os.path.getmtime(path)
            if indexed.get(path) == mtime:
                continue                           # unchanged -> skip the work
            writer.update_document(
                path=path, mtime=mtime,
                title=os.path.basename(path),
                content=extract_text(path),
            )
            updated += 1 if path in indexed else 0
            added   += 0 if path in indexed else 1

    # Files that vanished from disk should leave the index too
    for path in indexed:
        if path not in on_disk:
            writer.delete_by_term("path", path)
            removed += 1

    writer.commit()
    return added, updated, removed

Running it twice shows the incremental behaviour clearly:

>>> index_folder("docs", "kbindex")
(3, 0, 0)        # first run: 3 files added
>>> index_folder("docs", "kbindex")
(0, 0, 0)        # nothing changed: no work done
# ...edit one file, add one, delete one, then:
>>> index_folder("docs", "kbindex")
(1, 1, 1)        # 1 added, 1 updated, 1 removed

Those exact numbers came out of the script above against whoosh3 3.16.4 — the mtime check makes re-runs a no-op, so you can call this on a timer or a file-watch event cheaply.

5. Search with highlighted snippets

Search both the title and the body with a MultifieldParser, and use a ContextFragmenter so results come back as short highlighted excerpts instead of whole documents.

from whoosh.qparser import MultifieldParser
from whoosh.highlight import ContextFragmenter

def search(index_dir, query_str, limit=5):
    ix = index.open_dir(index_dir)
    parser = MultifieldParser(["title", "content"], schema=ix.schema)
    query = parser.parse(query_str)
    with ix.searcher() as s:
        results = s.search(query, limit=limit)
        results.fragmenter = ContextFragmenter(maxchars=120, surround=40)
        for hit in results:
            print(f'{os.path.basename(hit["path"])}  (score {hit.score:.2f})')
            print("   ", hit.highlights("content"))

>>> search("kbindex", "server")
notes.txt  (score 1.44)
    we chose Whoosh because it needs no <b class="match term0">server</b> and is easy...

Why Whoosh for this

• No server. The index is a folder of files; ship it with your app or in a Docker image.
• Pure Python. No native builds, so it installs cleanly in slim containers, CI, and even AWS Lambda layers.
• Real ranking. BM25F scoring, phrase queries, fielded search, and highlighting out of the box.
• It scales down. For thousands to low-millions of documents on one machine this is often all you need.

pip install whoosh3

Related guides

• Whoosh for RAG: hybrid keyword + vector search
• Performance tuning: why indexing is slow and how to fix it
• Search files from the command line
• Autocomplete and prefix search

The indexing counts and the search output above came from running the exact code in this guide against whoosh3 3.16.4 before publishing; PDF and Markdown extraction quality depends on the extractor library and your files. "Whoosh" is the pure-Python search library originally by Matt Chaput; this is an actively maintained continuation.