Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "metric": "token_delta",
    "construct": "X proxy(<M>)",
    "models": [
        "tiktoken/cl100k_base@vocab",
        "tiktoken/o200k_base@vocab"
    ],
    "tokenizers": [
        "cl100k_base",
        "o200k_base"
    ],
    "estimand": {
        "population": {
            "description": "Agent status assertions whose only directly verified evidence is an adjacent measured quantity (a proxy for the asserted state).",
            "items_sha256": "b678ad9e0d697fe4ce5b91b28c06ebeef3df0ca9856f2285d65c0d4156856c2a"
        },
        "baseline": "Full careful English stating the assertion X, the directly verified quantity M, M's proxy status, and that the M-to-X inference is unverified — the proposal's own lossless mapping.",
        "aggregation": "Equal weight per pair; arithmetic mean per tokenizer; least-favourable tokenizer mean as the headline."
    },
    "design": {
        "items": 8,
        "balance": "8 distinct claim classes, one pair each",
        "selection": "All pairs and weights fixed before tokenization. Fresh domains; no text shared with the proposal's examples."
    },
    "test_set": [
        {
            "claim_class": "health",
            "english": "The service is healthy; what I directly verified is that the health endpoint returned 200, which is a proxy for health — the inference from a passing probe to a healthy service is unverified.",
            "ainglish": "The service is healthy proxy(<health-endpoint-200s>)."
        },
        {
            "claim_class": "correctness",
            "english": "The change is correct; what I directly verified is that the test suite passed, which is a proxy for correctness — the inference from green tests to a correct change is unverified.",
            "ainglish": "The change is correct proxy(<suite-green>)."
        },
        {
            "claim_class": "recoverability",
            "english": "The data is recoverable; what I directly verified is that the backup job exited zero, which is a proxy for recoverability — the inference from a clean exit to a restorable backup is unverified.",
            "ainglish": "The data is recoverable proxy(<backup-exit-0>)."
        },
        {
            "claim_class": "completion",
            "english": "The migration work is complete; what I directly verified is that the queue is empty, which is a proxy for completion — the inference from an empty queue to finished work is unverified.",
            "ainglish": "The migration work is complete proxy(<queue-empty>)."
        },
        {
            "claim_class": "improvement",
            "english": "The model improved; what I directly verified is that training loss fell, which is a proxy for improvement — the inference from lower loss to a better model is unverified.",
            "ainglish": "The model improved proxy(<train-loss-down>)."
        },
        {
            "claim_class": "billing",
            "english": "The customer was billed; what I directly verified is that the invoice email was accepted by the relay, which is a proxy for billing — the inference from relay acceptance to a delivered bill is unverified.",
            "ainglish": "The customer was billed proxy(<relay-accepted>)."
        },
        {
            "claim_class": "readability",
            "english": "The module is readable; what I directly verified is that the linter reported no findings, which is a proxy for readability — the inference from a clean lint to readable code is unverified.",
            "ainglish": "The module is readable proxy(<lint-clean>)."
        },
        {
            "claim_class": "adoption",
            "english": "The feature is adopted; what I directly verified is that the flag-enabled cohort grew, which is a proxy for adoption — the inference from cohort growth to genuine use is unverified.",
            "ainglish": "The feature is adopted proxy(<cohort-growth>)."
        }
    ],
    "method": "For each named tokenizer, len(encode(ainglish)) - len(encode(english)) per fixed pair; arithmetic mean; report the larger (least favourable) tokenizer mean.",
    "analysis_plan": "File the fixed result whether favourable or not. Per-pair and per-tokenizer cells preserved. This cost original makes no comprehension claim; the proposal's declared primary is a comprehension panel this row does not supply.",
    "seed": "none - deterministic tokenization"
}