The coin-flipping trick that counts the Internet

Counting page loads is simple, but counting distinct visitors precisely requires storing and checking every person’s identifier. At Internet scale, keeping separate rosters for millions of pages, ads or social media posts consumes enormous amounts of memory. HyperLogLog trades exact counts for highly accurate estimates, using very little storage. Its core idea comes from probability: in repeated coin flips, longer streaks of tails become increasingly rare, so the longest streak observed offers a clue to how many rounds were played. If a large audience produces a long streak, the record suggests many visitors; repeated visits do not change the count because each person generates the same result each time.

In practice, a hash function turns each visitor’s unique identifier into a seemingly random string of zeroes and ones, which the algorithm treats like coin flips. Rather than rely on one longest streak - which could be distorted by an unlikely outlier - HyperLogLog divides visitors into many groups based on the initial bits, estimates each group’s size from the remaining bits, and combines the estimates. Real systems use thousands of groups and a harmonic mean to limit the influence of extreme results. The resulting estimates are typically within 1 percent of the true count, while requiring only modest memory even for billions of users. The same technique supports website analytics, cybersecurity monitoring for unusual activity such as bot swarms, and biological comparisons of genomic data.