A Conversation with Paul Masurel, Creator of Tantivy

Paul Masurel created Tantivy in 2017 after working on search at Exalead and Indeed. He began the Rust project as a way to test his new programming language and deepen his understanding of search engines, drawing its architecture from Apache Lucene. His aim was not to replace Lucene feature for feature, but to offer a compact, modular library that companies could adapt with their own tokenizers and query parsers. Tantivy handles indexing and search while leaving deployment and coordination to the systems that embed it - a design that later made it useful for products with very different needs.

Tantivy’s early performance results, including benchmarks showing it often running twice as fast as Lucene, prompted a productive exchange between the projects. Lucene developers investigated the results and incorporated optimizations inspired by Tantivy; Masurel says Lucene now outperforms Tantivy in many areas, and both benefited from the collaboration. He rejects the idea that open-source competition must become a feud. Tantivy now underpins projects including Quickwit, ParadeDB, LNX, and Turso, while its growth makes contribution review and preserving simplicity ongoing challenges. Masurel describes deciding what to merge as the hardest part of maintenance, made tougher by limited time after Quickwit’s acquisition by Datadog. He also emphasizes that Tantivy’s progress has depended on a broad community of contributors, not just its creator.