Bag of Decisions Reranker

The piece proposes treating queries not as bags of terms but as "bags of decisions": use an LLM to generate a per-query rubric of yes/no relevance criteria (example: questions about desks designed for children, age ranges, ergonomics, buying guides). Each criterion becomes a noul question evaluated by a probabilistic decision model (Jev), which returns P(yes) for each question. Summing those probabilities produces a rerank score that can be used alone or combined with BM25. The workflow is illustrated with a short code sketch showing how to call the decision model for each question and aggregate probabilities into a final relevance score.

An experiment compares BM25, a single-question Jev reranker, and the bag-of-decisions reranker on two e-commerce datasets (Wayfair WANDS and Amazon ESCI). Mean NDCG: WANDS - BM25 0.5407, Jev 0.574, bag_of_decisions 0.610; ESCI - BM25 0.289, Jev 0.314, bag_of_decisions 0.360. Median NDCG improved notably on ESCI (0.171 → 0.341). The method functions as per-query feature engineering: LLMs generate diverse, query-specific signals and the decision model measures coverage; results outperform baselines and justify further work on prompt engineering and domain-focused decision models.