hn.today

Gateway-Free Kafka Streaming from FPGA-Based Detectors on Zynq-7000

arxiv.org10 points0 comments
Screenshot of Gateway-Free Kafka Streaming from FPGA-Based Detectors on Zynq-7000

The work demonstrates that an FPGA-based detector can send event data directly to a standard Apache Kafka broker without a separate gateway computer. On a Zynq-7000 system-on-chip, detector events are generated in programmable logic, passed to the processor through AXI4-Stream and DMA, then packaged as Kafka ProduceRequests and sent over TCP using lwIP. A configurable pipeline allows multiple Kafka requests to remain outstanding at once, with the aim of increasing throughput while tracking the timing of data transfer, publication, and broker acknowledgements.

The evaluation used two independent DMA sources and tested pipeline depths of 1, 2, 4, and 8. Across 80 runs, the system recorded 400,000 events; every run preserved the expected frame-counter sequence, and no Kafka errors were recorded. Throughput rose from about 5,300 events per second at depth 1 to 16,200 at depth 4, reaching roughly 16,800-17,500 at depth 8. The median time from publication to acknowledgement improved from about 1.6 milliseconds at depth 1 to 0.66 milliseconds at depth 4, then increased slightly at depth 8. These results identify depth 4 as a strong balance of throughput and acknowledgement latency, with a 95th-percentile interval of 0.81-0.83 milliseconds.

Read on arxiv.org0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Hardware

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.