Show HN: Spinf-serve – inference server for bulk scoring and decisions

Spinf-serve is an open-source inference server for bulk classification, extraction, and fixed-choice decisions, rather than conversational chat. It reads each piece of content once and scores many questions about it in parallel, returning probabilities for answer options instead of generated prose. Those scores include a baseline measured on empty content, making it possible to see how much the content changes an answer; users can set their own decision thresholds. The same pass can also extract text, numbers, or amounts, and supported models can process images, audio, and video. Prompt packs bundle question sets, scoring rules, and decision logic for tasks such as moderation, triage, and review analysis.

The project targets workloads with many questions per item, such as screening millions of news stories. Its benchmark on an RTX 4090 with Gemma 4 12B scored 1,344 questions across 24 articles in 2.7 seconds, compared with 6.7 seconds for vLLM using prefix caching; the advantage comes from branching questions from a shared content read and batching them into fewer forward passes. The server provides an OpenAI-compatible API and can run locally under Apache 2.0, though the recommended setup requires Linux, Python 3.12, CUDA 13 support, and a 24 GB NVIDIA GPU. Available checkpoints vary in size and supported inputs, and other models from supported architecture families can be loaded.