TrnRT is an Apache-2.0 open-source inference engine for running large language models on AWS Trainium2. It currently supports Z.ai’s GLM-5.3-Flash, a 320-billion-parameter mixture-of-experts model with 18 billion active parameters and a claimed context window of up to one million tokens. Its design uses custom Neuron Kernel Interface megakernels to combine decode operations and reduce kernel and communication overhead. A two-engine setup serves requests through an OpenAI-compatible API, while a session-aware router helps retain reusable prefixes in a cache spanning device and host memory. Model graphs are compiled offline on an x86 host, then loaded onto the Trainium server.
Benchmarks replaying coding-agent sessions report 3.57 million input and output tokens per minute on a single trn2.48xlarge at batch size 12, with 32 concurrent agents and a median of 22.3 decode tokens per second per user. The project estimates a cost of four cents per million tokens at the cited spot price, or 16.7 cents on a capacity block. These results came from an earlier build; the current repository’s checks report equivalent outputs for 384 requests and graph equivalence across 157 compiled graphs. Running the system requires a Trainium2 instance, and compiling the cache takes hours. GLM-5.3-Flash is the only supported model so far, with additional models planned.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.