Disk I/O profiling with eBPF

The profiling method measures disk activity at the Linux block layer rather than tracking every time a thread is switched off the CPU. eBPF hooks record a request’s submitting process and user-space stack, then match its completion to calculate I/O time. This captures synchronous and asynchronous operations, including libaio, io_uring, page-cache misses, direct I/O, and fsyncs. Each stack is associated with three measures: time spent on I/O, bytes read, and bytes written. Together, they distinguish a slower device from an application doing more disk work, and show which functions are responsible. Because the data uses the existing CPU-profile storage format, it adds little storage cost.

To limit overhead, the profiler takes at most one stack sample per CPU per millisecond and uses that sample to estimate nearby requests’ counts, bytes, and latency. Totals remain accurate, while attribution is approximate; kernel writeback performed on an application’s behalf is not attributed to the original writer. Tests found negligible impact on PostgreSQL workloads and high-rate fio, with roughly 1 MB of additional agent memory. A retention job on a MySQL cluster demonstrates the diagnostic value: deleting old rows increased disk activity, doubled query latency, and slowed a service by about 20%. Flame graphs separated the writer’s DELETE work from replica appliers and traced reads to cold pages fetched during index scans and row deletion. The profiles can be compared with a pre-incident period and linked to volume charts.