SIMD-accelerated computer vision on a $2 microcontroller

The ESP32-S3 is a roughly $2 dual-core microcontroller running at up to 240 MHz, with a small set of 128-bit SIMD instructions that are little known outside specialist libraries and examples. The project uses those instructions to accelerate the FAST computer-vision feature detector, which identifies image corners. The author describes learning the chip’s assembly and building a basic register allocator, basm, while developing the implementation. The goal was to process groups of 16 pixels at once, speeding up FAST’s preliminary test for whether a pixel might be a corner.

Two quirks of the processor shaped the implementation. It cannot directly load misaligned pixel blocks, so the code combines neighboring register data and extracts the needed east and west pixel slices. Its SIMD comparisons work only on signed 8-bit values, while image pixels are unsigned; moreover, SIMD arithmetic saturates, preventing a straightforward subtraction of 128. The solution is to XOR each pixel with 0x80, which remaps unsigned values from 0-255 into the signed range without arithmetic. These workarounds enabled vectorized comparisons and a similarly accelerated scoring function. In testing, throughput rose from 5.1 to 11.2 megapixels per second - about 2.2 times the reference implementation - and a QVGA frame takes roughly 6 milliseconds. The performance is sufficient for real-time VGA processing at 30 frames per second, showing that inexpensive microcontrollers can handle practical computer-vision workloads.