The BM25 Weighting Scheme

BM25 is presented as the default "Best Match" probabilistic term-weighting scheme used by Xapian and shown in TREC evaluations to outperform other known probabilistic schemes. The weighting tradition begins with the Robertson-Sparck Jones formula that multiplies a within-query-frequency factor, a within-document-frequency factor, and an IDF-like log term built from r, R, n, and N, with normalization by document length L and constants k1, k3 (and optionally k2 for query-length effects). BM11 adds an extra query-length-dependent term that vanishes when L = 1; BM15 alters the document-frequency normalization. BM25 unifies BM11 and BM15 by interpolating with a parameter b, producing K = k1(bL + (1 − b)), and abandons non‑unit power terms on f and K as unhelpful.

Xapian’s implementation uses the BM25 formula with practical choices: default parameters k1 = 1, k2 = 0, k3 = 1, b = 0.5; a floor on L (0.5) to prevent tiny documents receiving excessive weight; and a positive-adjusted extra term expressed as (2 k2 n_q)/(1 + L). Scaling factors such as (k1 + 1) and (k3 + 1) are retained to set sensible baselines. The scheme is presented as effective out of the box but tunable - optimal k and b values depend on the document collection and query types.