What BM25Engine Does

BM25Engine implements BM25 lexical scoring inside an ODC app. It tokenizes text, stems tokens, and scores candidate chunks against a query using the BM25 formula. It does not perform vector search, hybrid retrieval, RRF fusion, reranking, caching, or synonym expansion. Persistence, entity ownership, and workflow orchestration are the consuming app's responsibility.

Server Actions

TokenizeText

• Input: raw text string (chunk text at ingestion, or query text at search time)
• Output: list of stemmed tokens
• Pipeline: lowercase, replace non-alphanumeric characters with a space, split on whitespace, drop single-character pure-numeric tokens, remove stop words, Porter stem
• Must produce identical output at ingestion time and query time. Any divergence causes silent recall failures with no error raised.

IndexChunk

• Input: raw chunk text string
• Output: ChunkIndexResult containing TokenCount (Integer) and a list of TermFrequency items (Term, Frequency), one per unique stem
• Does not write to any entity. All persistence is the consuming app's responsibility.

ScoreQuery

• Input: list of stemmed query terms; list of PostingCandidate items (ChunkId, Term, TermFrequencyInChunk, DocumentFrequency, ChunkTokenCount); corpus stats (TotalChunks, avgChunkLength); tuning parameters (K1, B)
• Output: list of BM25ScoreResult items (ChunkId, Score), sorted descending
• Uses the Lucene IDF variant: IDF(t) = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1). This is always positive, avoiding negative score contributions from terms common in the corpus.
• DocumentFrequency is read once per unique term, not once per posting row.

Installation

1. Install BM25Engine from the Forge as an External Logic dependency in your ODC app.
2. Add the following entities to your ODC app:
   • BM25Term: Id, Term (Text, unique index), DocumentFrequency (Integer)
   • BM25Posting: Id, BM25TermId (FK, indexed), DocumentChunkId (FK, indexed), TermFrequency (Integer)
   • BM25Stats: Id, TotalChunks (Integer), TotalTokenCount (Long Integer)
3. Add a TokenCount (Integer) attribute to your existing DocumentChunk entity.
4. Add three Site Properties to your app:
   • K1 (Decimal, default 1.2)
   • B (Decimal, default 0.75)
   • TokenizerVersion (Text, default v2-stemmed-porter)

Configuring the Ingestion Workflow

Create an IndexChunks Server Action or Workflow that runs after embedding. For each DocumentChunk row:

1. Call BM25Engine.IndexChunk with the chunk text.
2. Write the returned TokenCount back to the DocumentChunk row.
3. Iterate the returned TermFrequency list. For each term:
   • Find or create a BM25Term row for the stemmed term.
   • Create a BM25Posting row with BM25TermId, DocumentChunkId, and TermFrequency.

After all chunks are indexed, trigger RecomputeBM25Stats manually from the ODC Portal before running any search.

Configuring the RecomputeBM25Stats Timer

Create a Timer called RecomputeBM25Stats in your app. Its action must:

1. Run COUNT(DISTINCT DocumentChunkId) GROUP BY BM25TermId over BM25Posting and update DocumentFrequency on each BM25Term row.
2. Write or update the BM25Stats singleton row with TotalChunks and TotalTokenCount computed from the current state of BM25Posting.

This timer must run at least once after the first index. If BM25Stats has no row, ScoreQuery divides by zero. Schedule it to run periodically to keep DocumentFrequency current after new ingestion.

Using BM25Engine at Query Time

1. Call BM25Engine.TokenizeText on the user query to get the stemmed query terms.
2. Look up each stemmed term in BM25Term to retrieve its DocumentFrequency.
3. Fetch all BM25Posting rows matching those terms to build the PostingCandidate list.
4. Retrieve the BM25Stats singleton for TotalChunks and TotalTokenCount.
5. Call BM25Engine.ScoreQuery with the query terms, posting candidates, corpus stats, and tuning parameters (K1, B).
6. Use the returned BM25ScoreResult list to fetch and rank matching DocumentChunk rows.

Tokenizer Version Management

The TokenizerVersion Site Property (default v2-stemmed-porter) tracks which tokenizer pipeline produced the current index. BM25Engine does not detect version mismatches. If the tokenizer changes in a future release:

1. Update the TokenizerVersion Site Property to match the new version.
2. Set TokenCount to null on all DocumentChunk rows.
3. Delete all BM25Posting, BM25Term, and BM25Stats rows.
4. Re-run IndexChunks across the full corpus.
5. Trigger RecomputeBM25Stats manually.

Skipping the reindex after a tokenizer change causes query stems to stop matching indexed stems. Recall degrades silently with no error.

BM25 Tuning Parameters

• K1 (default 1.2): controls term frequency saturation. Higher values increase the reward for repeated term occurrences in a chunk.
• B (default 0.75): controls length normalisation. Set to 0 to disable length normalisation. Set to 1 for full normalisation where longer chunks are proportionally penalised.

Scale Boundary

ScoreQuery performs a linear scan over the candidate posting set. The scale ceiling has not been load-tested. Posting fan-out for common terms will eventually become the bottleneck. Run load tests against your own corpus size before using this in production at scale.