Simon Willison Updates ttok Tool for GPT-6 Tokenization

Independent developer Simon Willison has officially released version 1.0 of ttok, a command-line utility designed to count and truncate text based on tokens. The major milestone release shifts the tool's default settings to align with OpenAI's latest generation of large language models, moving away from the older GPT-4 standard to better serve modern AI engineering workflows.

The decision to push the tool to its 1.0 version came after Willison upgraded from ttok 0.4 using the command "uv tool upgrade ttok" and noticed the utility was still defaulting to the GPT-4 tokenizer. Recognizing that modern workflows require compatibility with newer architectures, he updated the default to target the GPT-5 and GPT-6 families instead, providing a reasonable excuse to finally ship a stable 1.0 release.

While OpenAI has not publicly confirmed that its GPT-6 models share a tokenizer with the GPT-5 family, Willison cited experimental evidence to justify the change. He pointed to a commit by developer William Liu, who tested seven distinct GPT models: GPT 5.5, GPT 5.6 Sol, GPT 5.6 Terra, GPT 5.6 Luna, as well as GPT 6 Astra, GPT 6 Sol, and GPT 6 Luna.

Liu's experiment revealed that all seven of these models reported exactly 44,794 tokens and matched perfectly across 31 different test fixtures. The findings indicate that GPT-6 introduces no input-count changes on this specific corpus, suggesting the underlying tokenization mechanics remain identical to its predecessor. This consistency allows ttok 1.0 to reliably serve developers working across both model generations, ensuring accurate token budgeting as context windows and API costs continue to evolve.