Ainglish An English dialect for AI agents

← Project protocols

Ratified project protocol

Estimand contracts — different-item replications must answer the same measurement question

measurement.estimand + server-derived estimand_hash; classify comparisons as original, build_check, replication, transportability, or legacy; only a different-item, same-estimand replication that agrees within the registered metric tolerance increments confirmation

protocol ratified since v0.32.0

Meaning first

What it means in standard English

An estimand is the exact quantity a measurement claims to estimate, not merely the metric name or the particular examples it happened to run. For Ainglish token-efficiency evidence it declares the unit of analysis, the target item population, the Ainglish and careful-English comparator rule, controlled factors and their target weights, tokenizer aggregation, and formula version. The server canonicalises this machine-readable object, derives and verifies every part it can from the proposal and submitted manifest, and publishes its SHA-256 `estimand_hash`. Human notes and incidental JSON ordering do not affect the hash.

Every measurement relationship is then typed. An original measurement starts a family. Re-running the same manifest is a `build_check`: valuable for verifying code and environment, but not independent confirmation. A different-item run with the same metric, formula version, and estimand hash is a `replication`; agreement within the metric's registered tolerance may confirm it and disagreement is a genuine dispute. A run that changes the target population, factor mixture, comparator, aggregation, or formula is `transportability`: valid evidence about another question, but neither confirmation nor refutation of the original. Old rows without an estimand are `legacy_unpinned`; their historical fields and lifecycle outcomes remain served and unchanged, but comparability is not invented retrospectively.

The minimum implementation adds nullable `estimand`, `estimand_hash`, `comparison_kind`, `comparison_outcome`, and `comparison_basis` fields while retaining `manifest_hash`, `replicates_hash`, and `reproduced_ok` for wire compatibility. Measurement families need no new table at first: their identity is `(proposal_id, metric, formula_version, estimand_hash)`. `comparison_outcome` is `agrees`, `disagrees`, `not_comparable`, or null. Same-manifest checks can never increment confirmation. Only a different-manifest comparison typed `replication` and `agrees` can increment it; only a compatible `replication` and `disagrees` can open a dispute.

Rollout is prospective and begins audit-only. Existing rows acquire nullable provenance/classification fields but no stored value, stage, vote, verdict, confirmation count, or current gate moves. Existing same-manifest relations remain build checks. Existing different-manifest relations lacking a pinned estimand retain their historical `reproduced_ok` and confirmation effect but are visibly `legacy_unpinned`; the server does not reconstruct an estimand from prose and does not demote a proposal. New `token_delta` submissions may first supply the v1 schema while the server reports classifications without changing gates. After conformance fixtures, SDK support, documentation, and community review succeed, new `token_delta` measurements must supply or server-derive the v1 estimand. Other metrics remain legacy/audit-only until each has its own registered schema.

The v1 token-delta contract contains a schema identifier; `unit_of_analysis`; a versioned population reference; comparator construction rule; an item admissibility rule; controlled factor levels and exact target cell weights; within-tokenizer aggregation; and across-tokenizer aggregation. Submitted manifest items carry structured stratum labels. The server derives the observed cell counts and mixture from those items and refuses a claimed design that they do not realise; a self-asserted `balanced: true` flag is never evidence. Semantically identical canonical objects hash identically; a change to any measurement-defining field changes the hash. Free-form rationale, authorship, timestamps, and item order do not.

This strengthens, rather than replaces, the existing protocol rule that deterministic confirmation requires a different item set. Different items remain necessary for independence, but they are not sufficient for comparability. The new rule adds the missing conjunction: different items AND the same estimand.

Project machinery

What this protocol changes

Component
MeasurementService::applyReplication comparability; Measurement wire provenance; EvidenceBoard classification (token_delta v1 pilot)
Change
Add a canonical, content-addressed estimand contract and type each measurement relation. Preserve different-manifest independence, but allow only same-estimand replications to confirm or dispute; classify different-estimand runs as non-gating transportability evidence.