hn.today

Not Everything Needs an LLM

newsletter.posthog.com173 points72 comments
Screenshot of Not Everything Needs an LLM

Decision models answer structured questions with typed outputs and calibrated probabilities, making uncertainty visible instead of producing open-ended prose. PostHog uses them for narrow judgments that are too fuzzy for rules or regexes but too simple to justify an LLM. In Replay Vision, two yes/no checks identify possible personal information in session summaries; only scores of 0.4 or higher trigger Gemini to rewrite a summary. This means 98.7% of summaries avoid an LLM call. Thresholds should reflect the cost of errors: a low threshold catches more sensitive data, while a high threshold helps prevent an unwanted Slack bot response. Decision models can also route requests to the cheapest suitable language model, rerank embedding-search results, and choose chart types from SQL and sample rows. Reranking improved quality by about 25% across 20 test queries.

These models are fast and inexpensive, but their probabilities do not eliminate mistakes, and thresholds vary across tasks and model versions. PostHog recommends calibrating with labeled examples, keeping models swappable, and using shadow mode when assessing changes. Decision models work best when they judge a clearly framed input in one pass; chaining questions or asking for complex, multi-step reasoning can create costly, slow workloads. A test ranking code-review findings barely outperformed chance, illustrating the limits of small-context evaluators on open-ended judgments. Language models remain the better choice for tasks requiring substantial context or reasoning, while hybrid decision models may cover some cases in between.

Read on newsletter.posthog.com72 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.