hn.today

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

github.com53 points7 comments
Screenshot of Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

TrnRT is an Apache-2.0 open-source inference engine for running large language models on AWS Trainium2. It currently supports Z.ai’s GLM-5.3-Flash, a 320-billion-parameter mixture-of-experts model with 18 billion active parameters and a claimed context window of up to one million tokens. Its design uses custom Neuron Kernel Interface megakernels to combine decode operations and reduce kernel and communication overhead. A two-engine setup serves requests through an OpenAI-compatible API, while a session-aware router helps retain reusable prefixes in a cache spanning device and host memory. Model graphs are compiled offline on an x86 host, then loaded onto the Trainium server.

Benchmarks replaying coding-agent sessions report 3.57 million input and output tokens per minute on a single trn2.48xlarge at batch size 12, with 32 concurrent agents and a median of 22.3 decode tokens per second per user. The project estimates a cost of four cents per million tokens at the cited spot price, or 16.7 cents on a capacity block. These results came from an earlier build; the current repository’s checks report equivalent outputs for 384 requests and graph equivalence across 157 compiled graphs. Running the system requires a Trainium2 instance, and compiling the cache takes hours. GLM-5.3-Flash is the only supported model so far, with additional models planned.

Read on github.com7 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.