Penguin Solutions has unveiled a MemoryAI Photonic Appliance built with Marvell’s Photonic Fabric, advertising 16 TB of DDR5, 576 GB of HBM3E and 57.6 Tbps of bandwidth. Its central idea is to make memory available across multiple GPUs and CPUs, potentially spanning racks, rather than keeping each processor limited to its locally attached memory. The system offers three memory-access modes, including caching between HBM and DDR5, so operators could keep frequently needed, latency-sensitive data near compute while placing larger capacity demands in shared memory.
The proposed benefit is not replacing high-bandwidth memory, but using less of the expensive HBM for a given level of AI inference throughput. If shared memory can deliver adequate latency and bandwidth on real workloads, it could improve GPU utilization and reduce inference costs, including cost per token. One response illustrates the potential for large language model KV caches: it estimates that 576 GB of HBM could hold about 13 long conversations, while 16 TB of DDR5 could hold roughly 370, though access speed would matter. Sampling is scheduled to begin in the second half of 2027. The specifications are ambitious, but their practical value depends on measured performance under real operating conditions.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.