Frontier quality at 1% of the cost: a production benchmark of LLM classifiers

Airbnb benchmarked a production model that detects violations of its pricing and payment transparency policy, where false positives can block legitimate listings and decisions must arrive in real time. On 10,000 weighted, held-out examples, the team compared a fine-tuned 2-billion-effective-parameter Gemma model with self-hosted open models and vendor APIs, using a shared concise prompt. Classification quality was measured by maximum F1, with thresholds swept where models returned scores; latency was measured at the serving gateway. The task runs thousands of times per minute, so both speed and reliable control over the precision-recall balance matter.

Fine-tuning on hundreds of thousands of domain examples raised the small model’s F1 from 0.563 to 0.902, matching leading proprietary models while returning scores in about 35 milliseconds median and 85 milliseconds at p99. Off-the-shelf small models were faster but less accurate; frontier APIs could match quality with a much longer detailed prompt, but were typically far slower and more expensive. At high traffic, one GPU cost roughly a hundredth as much as frontier APIs, while APIs were cheaper only below about 15-30 requests per minute. Score-based outputs also let teams adjust thresholds without rewriting prompts. The findings support fine-tuning when a task has high volume, tight latency requirements, labeled in-domain data, and a need to tune false-positive versus false-negative rates; prompts and frontier APIs remain practical for low-volume or rapidly changing tasks.