42x faster prompt lookup drafting in llama.cpp

This work accelerates prompt lookup (n‑gram) drafting in llama.cpp, achieving up to 42× faster token drafting and up to 2.6× lower memory use through a sequence of practical implementation changes. Benchmarks on WikiText‑103 static caches (sizes 0-541 MB) show upstream drafting latencies of 8.54-165.48 µs per drafted token collapsing to about 0.89-3.98 µs with all changes. Static cache load times and peak memory footprints also shrink markedly, with multi‑second loads reduced and gigabyte‑scale peaks lowered on large corpora.

Prompt lookup here uses three n‑gram caches (context, dynamic, static) and accepts a draft token y when s_n^f(y)=f(X_n,y)·w(y) meets count and fraction thresholds a_n and p_n (static entries get weight 100). The implementation optimizations are: stop copying inner maps (read them by reference), replace the outer std::unordered_map with ankerl::unordered_dense::segmented_map, and convert inner follower maps to sorted vectors because most n‑grams have very few followers (64% have one). The copy‑avoidance change alone delivered 4.5-25.6× speedups (drafting ≈1.9-6.5 µs); the flat hash map sped up loading and trimmed memory ~1.07-1.11×; combining all changes yields the reported 42× latency and multi‑fold memory reductions while keeping draft acceptance rates effectively unchanged.