Three mechanisms, three ledgers

Efficiency: now and later

Ainglish can be shorter than complete careful English today, yet unfamiliar to a reader today. Training a model may improve familiarity without changing literal token count. Training a tokenizer may change literal count without teaching the model how to use the distinction. This page keeps those claims separate and reports the adverse results beside the favourable ones.

Research snapshot: 28 August 2026. These experiments are product research, not ratification evidence.

The three ledgers

1. Surface encoding

What it counts: tokens in one fixed string under one named tokenizer.

What can change it: changing the string or training/adopting a different tokenizer.

What cannot: model-weight training. The model receives token IDs after segmentation.

2. Accommodation and selection

What it counts: glossary text and demonstrations needed before a reader understands and chooses the form.

What can change it: exposure in model training, retrieval or the prompt.

Boundary: exposure must be declared, not inferred from a model name.

3. Correct-outcome interaction

What it counts: every input and output token, retry and repair through a validator-accepted completion.

What can change it: comprehension, selection, verbosity, stopping policy and repair success.

Boundary: cost never travels without first-pass, eventual and unresolved-task counts.

What current tokenizers say

A reproducible census priced all 57 reviewed Ainglish ↔ careful-English pairs in the public v0.35.0 training pack. Negative means the Ainglish rendering used fewer tokens. These are complete reviewed pairs, not isolated display grammars.

Tokenizer	Mean delta, equal pairs	Pairs cheaper in Ainglish	Recent-use-weighted proxy
`cl100k_base`	−10.14	89.5%	−15.99
`o200k_base`	−10.04	89.5%	−15.56
`p50k_base`	−7.81	87.7%	−13.23

The first fixed-tokenizer exposure result

One experiment held the Qwen 2.5 7B base revision, 4-bit loading, tokenizer, 19 marker-free glosses and deterministic decoding fixed. The only condition change was a previously frozen two-epoch Ainglish LoRA. A wrong exact-form answer received one authoritative register repair.

This did not show that training exposure removes accommodation cost. Every item still needed a definition/repair turn. The adapter improved exact compliance after that repair, with five base-failure → adapter-success changes and none in reverse. The small token reduction came from shorter outputs and repeated history, not fewer turns. See the frozen prompts, outputs and receipt.

What tokenizer adaptation can change

A separate matched-budget simulation crossed 8k/16k BPE vocabularies, careful-English/Ainglish supplements and two pre-tokenization policies. Every cell received the same 8 MB English core plus a 2 MB supplement with each of the 57 semantic pairs repeated exactly 32 times; neutral filler equalised bytes.

Vocabulary	Hyphen handling	Ainglish release-token change	Whole markers gained	Held-out English change
8,000	punctuation split	−94	0	−164
8,000	hyphen unit visible	−368	10	+364
16,000	punctuation split	−93	0	−23
16,000	hyphen unit visible	−585	16	+237

The mechanism is visible: vocabulary exposure cannot create a whole-marker token when the pre-tokenizer forbids merges across hyphens. A hyphen-aware design can, but vocabulary slots have opportunity cost. This small simulation is not a prediction that a production tokenizer will choose the same trade-off. Inspect all eight tokenizers in the reproducible lab.

Where the boundary is enforced

A dated claim audit pinned 17 files across the website/server, language releases, SDK and both agent plugins. It found no statement that model-weight exposure changes a fixed tokenizer and no statement that publication proves training, adoption or benefit. It did retain one editorial risk: the broad “optimised for clearer and more efficient” mission line can sound like a completed result when detached from the methodology. Read the 17-file reproducible audit.

What follows

Now: compare complete Ainglish and careful-English utterances under named current tokenizers. Keep adverse rows visible.
For model builders: put the CC0 training pack into weight-training experiments, but use held-out tasks and report exposure composition.
For tokenizer builders: test marker-aware segmentation against a fixed general-English regression budget; do not optimise one vocabulary in isolation.
For project claims: call exposure unestablished unless a training receipt exists. Ordinary English has the incumbent-data advantage; that is a hypothesis boundary, not a waiver for adverse Ainglish results.
For flagship selection: avoid gratuitous segment count as a tie-breaker. Comprehension, ambiguity removed and robustness remain primary.

The next measurement

A discussed, server-preflighted draft would add an interaction_cost_delta receipt: paired tasks, a frozen validator, every token through a correct outcome, unresolved tasks, stopping/censor policy and a per-model exposure receipt. In response to public review, failure remains a co-primary count with no invented token penalty; a cheaper arm cannot carry support after failing its correctness gate, and traffic weighting remains a separate projection. The proposal is intentionally not filed while the discussion remains open. Read or challenge the public protocol discussion and its revised valid preflight receipt.