Project Sync: The journey to sample-perfect audio/video sync with WebCodecs

Project Sync addresses a subtle audio/video synchronization error in Mediabunny, a media-processing library built on WebCodecs. A single transcode can sound correctly aligned to a person, yet audio may lag video by tens of milliseconds; repeated transcodes compound the error until the delay becomes obvious. Human perception is more tolerant of audio lag than audio leading video, so casual listening can conceal the problem. The investigation traces the drift to AAC encoder priming delay: an encoder adds samples so it can reconstruct the beginning of the audio accurately, but WebCodecs does not report that delay or correctly represent it in output timestamps. The delay varies by encoder and platform; measurements found 2,112 samples for Apple’s AAC encoder at 48 kHz, or about 44 milliseconds.

Mediabunny’s solution is to measure each system’s encoder delay rather than rely on fixed values. It encodes an impulse - a signal with a single nonzero sample - and locates the resulting peak, then shifts encoded packet timestamps earlier by the measured amount. This preserves the audio while restoring its timing relative to video. The correction requires muxers to support negative timestamps, which the project implements differently for MP4/MOV, Matroska, HLS/MPEG-TS, and packed audio formats. The result is sample-level synchronization through repeated processing, within the limits of the codecs and playback systems involved. The work also highlights how a small API omission can undermine an otherwise functional media pipeline.