Agent-task benchmark

Sentence comprehension is not the end goal. The receiving agent has to choose the right action, avoid the wrong one and know when it needs clarification. This frozen benchmark tests that operational question against both ambiguous and equally explicit English.

Project-operated descriptive result, published 29 August 2026. “Cold” means absent from the served prompt; the models' actual pretraining exposure is unknown. Completely failed and schema-invalid calls remain in the denominators.

What happened

The primary comparator is careful English, because it carries the same complete source intent. A positive effect means Ainglish produced more zero-repair successes on the same frozen items; a negative effect means careful or bare English did better. These are reader-level descriptive effects, not confidence intervals or independent replications.

In the cold track, Ainglish trailed equally explicit careful English by 13.6 percentage points on the mean reader effect, while leading bare ambiguous English by 7.4 points. After one local definition, those contrasts were −3.5 and +16.5 points. The comparator changes the conclusion: beating an ambiguity baseline is useful, but does not show that the compact form is yet as usable as saying the full meaning in familiar English.

Reader-family robustness

Six of the 22 artifacts are Qwen variants, so a post-hoc audit also grouped the roster into 16 declared model lineages and gave each lineage equal weight. This reduces one obvious source of pseudo-replication; it does not make the lineages independent or turn a convenience roster into a model-population sample.

No leave-one-lineage-out calculation reversed any of the four aggregate directions. One supplied definition improved Ainglish's equal-lineage effect by +8.1 pp against careful English and +6.7 pp against bare English. Those ranges are exploratory stability summaries, not confidence intervals or population-level inference. Construct results were heterogeneous: the largest persistent deficit was true-as-worded / false-as-worded (-47.8 pp cold; -43.8 pp after one definition), with only two frozen items per construct. Inspect the deterministic post-hoc audit.

All-row immediate-action rates

Failures stayed visible

One reader, solar-pro:22b, returned HTTP 500 in all 132 cells. Another, deepseek-v2:16b, violated the exact decision schema in 127 of 132 first outputs. Neither was removed after the result was known. Invalid outputs across the full roster were common and similarly distributed among arms, so the run measures operational compatibility as well as semantic understanding.

Present interaction cost

No present token-efficiency win appeared: mean raw interaction tokens were 229 versus 223 in the cold track and 246 versus 223 after one exposure. The definition is correctly charged to Ainglish; provider token coverage was 462 / 484 rows in each group. These counts come from different model tokenizers, so they are descriptive—not one tokenizer-independent efficiency estimate or a forecast for models trained on Ainglish.

What this says about training exposure

Familiar English has an incumbent training-data advantage; Ainglish was not known to be present in these readers' training corpora. That makes future training exposure an important hypothesis, but does not make today's adverse comparison disappear. A separate 264-observation comparison held one Qwen 2.5 7B base model and tokenizer fixed and added a tiny existing Ainglish LoRA. It did not improve immediate Ainglish task execution, its final gains were not selective to Ainglish, and it used more interaction tokens in every group.

That adapter had only 76 training rows. It is not a test of including a release in foundation-model pretraining, nor of tokenizer integration. The future claim therefore remains open and testable: larger, declared training exposure may close the familiarity gap, but the project has not shown that yet. Inspect the controlled-exposure receipt.

A separate semantics stress test

A 540-cell adversarial gauntlet supplied each construct's exact meaning to 3 installed readers. Their accuracy ranged from 88.9–95.6%, but 19 of 38 errors occurred when the correct answer was “underdetermined.” The dominant failure was treating “not stated” as “stated false.” This is a documentation and regression signal, not cold-comprehension, human-intuitiveness or governance evidence. Inspect the frozen gauntlet.

The question it asks

A source intent is fixed outside the message. A receiving agent sees one wording and a small set of executable actions. It may act immediately or ask for clarification. If it asks, it receives one scripted repair turn that states the same source intent without changing the task.

One task, all three arms

Source intent: the receiving agent is included in the group expected to review the draft.

What gets reported

• Zero-repair task success: the first action set exactly matches a valid set.
• Final task success: the action is correct after at most one scripted clarification.
• Wrong-action rate: the agent acts, but its action set is unsafe, incomplete or excessive.
• Clarification and invalid-output rates: neither disappears from the denominator.
• Repair-inclusive tokens and latency: reported only with coverage, provider and tokenizer or usage source.

There is no composite score: shorter text cannot compensate for a wrong action. Results remain item-level and should be stratified by construct and reader family before anyone reports an overall figure.

Exposure tracks in the frozen design

A future model-adaptation track would need a new frozen task release, training receipts and held-out items. It cannot be silently added to this version.

Audit or run it

The result is pinned to commit bbea0f8528484f780e591a727241f0fbe54e8d62. Its raw-response SHA-256 is d23e8da6573e05fe1fe79c762ffb4bc7eaddd169feeff0a78319a91044b93b2f and its analysis SHA-256 is 78dacc62b827dd053a6e38c894a62b9e36e5d4f347fc8f34d5ec54598e86623d. The original protocol artifact remains pinned to commit 028052715cfa61744fab0ca92268f71073de2246 and register 0.35.0. The post-hoc lineage audit is pinned to commit 116d7bef0b423ce6c53811516288f9344df85708 with result SHA-256 5cb40b9e79ff6421ae361351791b7cadcaab648a451937252f472653621ae56e.

python3 benchmark.py validate
python3 benchmark.py export --track cold --arm all --seed 20260828 > prompts.jsonl
python3 benchmark.py score responses.jsonl

Read the research status, inspect the measurement methodology, or use the frozen packet to publish a result that disagrees with the project.