Gaussian Splats part 3: There's always a bigger scene

An Unreal Engine renderer for Gaussian splats initially appeared able to handle unlimited scenes, but it relied on GPU Scene, which caps instances of one mesh at about 16 million. Replacing that system with indirect instanced drawing removed the cap: splat data stayed in GPU textures, while a shader generated sort keys, culled splats, and set the draw count each frame. A test scene of roughly 30 million splats then rendered successfully, and GPU-driven visibility decisions created a useful foundation for level of detail. However, this solved how many splats could be drawn, not how many could fit in memory. At about 32 bytes per splat, large real-world captures exceeded available VRAM.

The solution is to load only the needed portions of a scene and use progressively decimated versions of each region. PlayCanvas’s streamed SOG format organizes regions into a spatial tree, with multiple detail levels per region. The renderer selects a level using screen-space error: it estimates how large splat spacing appears in pixels and chooses the coarsest level that stays within a quality threshold. Users can adjust that threshold through Low, Medium, and High settings. Chunks arrive asynchronously, coarse levels first, so scenes appear quickly and sharpen as finer data loads; careful synchronization prevents drawing data before GPU uploads finish, and selection must support multiple cameras. Testing an enormous capture exposed another limit: texture dimensions maxed out at 16,384, creating a new obstacle to scaling further.