hn.today

UTF-8000: Unlimited UTF-8

utf-8000.jb2170.com76 points51 comments
Screenshot of UTF-8000: Unlimited UTF-8

UTF-8000 is a proposal and reference implementation (pipx install UTF-8000) that extends UTF-8 to arbitrarily long code units by allowing multiple contiguous start bytes while preserving UTF-8's self-synchronization, ASCII subset, and other properties. It separates high bits of bytes into self-synchronization prefixes (0 for ASCII, 10 for continuation, 11 for multi-byte first bytes) and start bits that form a unary length code (n−2 ones followed by 0) which can be striped across start and continuation bytes. Bytes within a code unit can serve simultaneously as continuation and start bytes, enabling “straddled” content bits and very large encodings; a 22-byte example is presented as a prototypical demonstration. The project is standalone and not endorsed by the Unicode Consortium.

Bit accounting and anti-overlong rules are explicit: an n-byte code unit carries 5n+1 content bits for n ≥ 2 (ASCII is the single exception with 7 content bits), with mandatory content bits of 0 for ASCII, 4 for two-byte units, and 5 for longer units to forbid overlong encodings and preserve uniqueness. Error recovery and self-synchronization are retained for random access and stream robustness, some bytes like 0xC0/0xC1 remain permanently invalid, and the specification stresses predictable bit counts, clear bit allocation, and backward compatibility with UTF-8 decoding semantics while enabling arbitrarily large codepoints.

Read on utf-8000.jb2170.com51 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.