Current-tokenizer cost (Δ, worst tokenizer)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "metric": "token_delta",
    "construct": "X proxy(<M>)",
    "models": [
        "tiktoken/cl100k_base@vocab",
        "tiktoken/o200k_base@vocab"
    ],
    "tokenizers": [
        "cl100k_base",
        "o200k_base"
    ],
    "estimand": {
        "population": {
            "description": "Agent assertions whose only directly verified evidence is an adjacent measured quantity used as a proxy for the asserted state.",
            "items_sha256": "19793e93aa11b6e739fd857f7890b69d0502d3b7187c504dc0893a5f569762b2"
        },
        "baseline": "Full careful English stating assertion X, directly verified quantity M, M's proxy status, and that the M-to-X inference is unverified, following the proposal's lossless mapping.",
        "aggregation": "Equal weight per pair; arithmetic mean per tokenizer; least-favourable tokenizer mean as the headline."
    },
    "design": {
        "items": 10,
        "balance": "10 distinct fresh claim classes, one pair each",
        "selection": "All pairs and weights fixed before tokenization. No item text copied from the proposal examples or the original manifest."
    },
    "test_set": [
        {
            "claim_class": "security",
            "english": "The release is secure; what I directly verified is that the vulnerability scanner found no known issues, which is a proxy for security — the inference from a clean scan to a secure release is unverified.",
            "ainglish": "The release is secure proxy(<scanner-clean>)."
        },
        {
            "claim_class": "freshness",
            "english": "The index is current; what I directly verified is that its rebuild timestamp is recent, which is a proxy for freshness — the inference from a recent rebuild to a current index is unverified.",
            "ainglish": "The index is current proxy(<recent-rebuild>)."
        },
        {
            "claim_class": "reachability",
            "english": "The recipient is reachable; what I directly verified is that DNS resolution succeeded, which is a proxy for reachability — the inference from a DNS answer to a reachable recipient is unverified.",
            "ainglish": "The recipient is reachable proxy(<dns-resolved>)."
        },
        {
            "claim_class": "privacy",
            "english": "The dataset is private; what I directly verified is that its access policy names a restricted group, which is a proxy for privacy — the inference from the policy text to effective privacy is unverified.",
            "ainglish": "The dataset is private proxy(<restricted-policy>)."
        },
        {
            "claim_class": "fairness",
            "english": "The ranking is fair; what I directly verified is that group-average scores are equal, which is a proxy for fairness — the inference from equal averages to a fair ranking is unverified.",
            "ainglish": "The ranking is fair proxy(<equal-group-means>)."
        },
        {
            "claim_class": "latency",
            "english": "The interaction is responsive; what I directly verified is that median latency is low, which is a proxy for responsiveness — the inference from a low median to a responsive interaction is unverified.",
            "ainglish": "The interaction is responsive proxy(<low-median-latency>)."
        },
        {
            "claim_class": "intent",
            "english": "The user approved the change; what I directly verified is that an approval button was clicked, which is a proxy for informed intent — the inference from a click to informed approval is unverified.",
            "ainglish": "The user approved the change proxy(<approval-click>)."
        },
        {
            "claim_class": "deployment",
            "english": "The deployment succeeded; what I directly verified is that the orchestrator marked the rollout complete, which is a proxy for success — the inference from rollout completion to a successful deployment is unverified.",
            "ainglish": "The deployment succeeded proxy(<rollout-complete>)."
        },
        {
            "claim_class": "integrity",
            "english": "The archive is intact; what I directly verified is that its checksum matches the stored digest, which is a proxy for integrity — the inference from one matching digest to an intact archive is unverified.",
            "ainglish": "The archive is intact proxy(<checksum-match>)."
        },
        {
            "claim_class": "consensus",
            "english": "The team agrees; what I directly verified is that nobody objected during the review window, which is a proxy for consensus — the inference from silence to agreement is unverified.",
            "ainglish": "The team agrees proxy(<no-review-objection>)."
        }
    ],
    "method": "For each named tokenizer, len(encode(ainglish)) - len(encode(english)) per fixed pair; arithmetic mean; report the larger (least favourable) tokenizer mean.",
    "analysis_plan": "File the fixed result whether favourable or not. Preserve every per-pair and per-tokenizer cell. This cost replication makes no comprehension claim.",
    "seed": "none - deterministic tokenization"
}