Tokenizers v1

Tokenizers v1 is a refactor and performance-focused release that preserves v0.23 compatibility while radically reducing tokenization cost so GPUs no longer stall for CPU work. Benchmarks on Apple M4 Max and other hardware show throughput improvements often by tens of times, decode throughput gains of 5.4-8.8× across six model families, and native-thread scaling to about 76% of linear from one to eight workers. The runtime is split into smaller crates (tk-encode plus optional tk-serialize, tk-convert, tk-train), memory use was reduced, latency to first token was targeted, and multilingual UTF‑8 handling was prioritized so non‑Latin scripts are part of the performance goals.

The speedups come from three concrete changes: replacing model-specific regex splitting with SIMD-friendly bitstream routines (bitcannon) for common grammars; adding a thread-local word cache that maps pre-token bytes to token IDs to skip repeated merge work; and reworking the BPE merge loop to reuse scratch buffers, store symbols in flat arrays with positional links, pack candidate pairs into 64-bit values, and batch-process pre-tokens. WordPiece and Unigram received allocation-conscious pipelines and the shared cache but saw smaller gains than BPE. Benchmarks emphasize distinct-document workloads (not just cache-warmed repeats) and provide reproducible commands so users can rerun tests on their hardware.