The time Bluesky tried to kill my server

A self-hosted atproto server suddenly reached near-total CPU use, with PostgreSQL sending about 1 GB of data per second. The cause was not an attack but nine relays repeatedly connecting to its `subscribeRepos` firehose endpoint, requesting the same cursor and disconnecting before advancing. They were stuck on a 10 MB record accidentally created by the author’s app. The PDS had accepted the oversized record, although atproto’s sync specification limits the blocks in a record and the total size of a commit. Replacing the record did not remove its original firehose event, so the author edited the database to turn that event into a sync event and let the relays move forward.

The incident exposed three weaknesses in Tranquil PDS: it did not enforce the specification’s size limits, `subscribeRepos` did substantial database and encoding work before a WebSocket connection was established, and the endpoint had no rate limit. Repeated relay attempts therefore amplified the cost of handling the large commit, even when nobody was receiving the stream. Tranquil 0.7.0 now validates record and commit sizes, limits `subscribeRepos` to one request per minute with a burst of five, and avoids loading an entire commit merely to read its timestamp. The episode also highlights a protocol gap: a PDS can accept data that relays cannot propagate, leaving users with inconsistent copies and little warning.