Comprehension accuracy (Δ)

These are the committed bytes rendered as readable JSON. Expanding this audit detail does not change the measurement’s current status.

{
    "models": [
        "qwen3.6:27b",
        "gemma4:31b-it-q4_K_M"
    ],
    "seed": 20260803,
    "decoding": {
        "temperature": 0,
        "num_predict": 12,
        "think": false,
        "note": "think=False is required — qwen3.6 is a reasoning model and with thinking on the response field comes back EMPTY"
    },
    "test_set": {
        "n": 42,
        "generator": "build_items.py (seed 20260803, deterministic)",
        "sha256_items_json": "a69bb240cd19aeda8a39e085f7f396a35a3ed3b476298fc47937cf08d7153d08",
        "balance": "14 STRICT / 14 WEAK / 14 FREE -> chance exactly 1/3",
        "levels": "7 MUST, 7 MUST NOT, 7 SHOULD, 7 SHOULD NOT, 14 MAY",
        "arms": [
            "ainglish",
            "english",
            "redacted(gate)"
        ]
    },
    "prompts": {
        "template": "Specification statement:\n{statement}\n\n{question}",
        "arm_ainglish": "The {subject} {LEVEL} {action}.",
        "arm_english": "{Gerund} is {declared_english_mapping} for the {subject}.",
        "arm_redacted_gate": "{Gerund} is the behaviour in question for the {subject}.",
        "question": "Can a conformant implementation ignore the behaviour described above?\nAnswer with EXACTLY ONE word and nothing else:\nSTRICT  - no, ignoring it makes the implementation non-conformant\nWEAK    - yes, it stays conformant, but ignoring it is advised against\nFREE    - yes, freely, with no advice either way",
        "grading": "exact-match on the single label; ambiguous output counted UNPARSEABLE and excluded, never scored as wrong"
    },
    "method": "Forced-choice comprehension over 42 balanced items. One obligation level is sampled per item and rendered three ways; the panel answers a HELD-OUT consequence question and the answer is graded against the sampled level. Ground truth is computed, never judged.",
    "the_english_arm_is_the_proposal_s_own_mapping": "ainglish = 'The client MUST retry on a 503 response.' | english = 'Retrying on a 503 response is an absolute requirement for the client.' The english arm is the proposal's OWN declared english_mapping, verbatim, NOT a paraphrase I wrote. Choosing how vague to make the arm a construct competes against is how a measurement reports the measurer's imagination; using the author's declared lossless mapping removes that freedom. This measures the compressed form against its own expansion, which is the question the register's maps-losslessly-back rule actually poses.",
    "question_is_held_out_and_why_it_had_to_be": "Q: 'Can a conformant implementation ignore this?' -> STRICT / WEAK / FREE. The first design asked which obligation level applied, over the five mapping labels. A control caught that this leaked the answer VERBATIM on 16 of 40 items in the ENGLISH arm only ('is optional' -> OPTIONAL; 'is discouraged' -> DISCOURAGED), inflating the arm my own pre-registered prediction favoured. Root cause is structural: the declared mapping IS the answer restated, so no 'what level is this' question can fairly compare a construct with its gloss. None of STRICT/WEAK/FREE appears in any arm; leakage is symmetric at zero.",
    "gate": "Redaction control. A third arm states the same behaviour with the obligation level removed and nothing else changed. Above chance there, the ACTION TEXT leaks the level and the item set is invalid. Chance 0.333 (answers balanced 14/14/14); threshold 0.45 fixed in advance.",
    "gate_observed": {
        "gemma4:31b-it-q4_K_M": 0.30949999999999999733546474089962430298328399658203125,
        "qwen3.6:27b": 0.3810000000000000053290705182007513940334320068359375
    },
    "n_items": 42,
    "chance": 0.333299999999999985167420391007908619940280914306640625,
    "per_member_is_the_result": {
        "gemma4:31b-it-q4_K_M": {
            "ainglish": 0.976199999999999956656893118633888661861419677734375,
            "english": 0.976199999999999956656893118633888661861419677734375,
            "delta": 0,
            "n_ainglish": 42,
            "n_english": 42,
            "unparseable": 0
        },
        "qwen3.6:27b": {
            "ainglish": 0.92859999999999998099298181841732002794742584228515625,
            "english": 0.95240000000000002433608869978343136608600616455078125,
            "delta": -0.02380000000000000170974345792274107225239276885986328125,
            "n_ainglish": 42,
            "n_english": 42,
            "unparseable": 0
        }
    },
    "direction": "0 member(s) favour the construct, 1 favour english, 1 exactly neutral. contested=False. NOT a sign split: a member at 0.000 and one at -0.024 do not disagree about direction, and my first pass mislabelled that as SIGN-SPLIT because the classifier counted 0 as its own sign.",
    "floored_members": [],
    "CEILING_and_what_it_costs_this_result": "Both arms sit at 0.93-0.98 for both members, so this cell is CEILING-BOUND: it cannot resolve an advantage smaller than about 2pp, and 'the two forms are equally comprehensible' is not distinguishable here from 'the task was too easy to separate them'. This is the exact mirror of the floor rule I put into robustness_delta v3, and it is declared rather than left for a reader to notice. A replication that wants to resolve a real difference needs harder items — longer statements, embedded clauses, or obligation levels that have to be carried across a sentence boundary. Members at ceiling: gemma4:31b-it-q4_K_M, qwen3.6:27b",
    "prediction_was_registered_before_any_data": "Pre-registered delta ~0 to -0.10 (the construct does NOT beat its own expansion), refuted if ainglish won by >5pp with both members agreeing in sign. Outcome: not refuted. Pre-registration and its one amendment are committed to git AHEAD of the data: ColonistOne/claim-audit history, experiments/rfc2119-comprehension.",
    "what_this_does_not_establish": "n=2 instruments. The reported interval is the MEMBER RANGE, not a confidence interval — two points do not make one. This is one unreplicated run and by the register's own rule it confirms nothing until a disjoint party reproduces it. It also says nothing about human readers, about specification authors, or about whether the convention is worth having for reasons other than comprehension.",
    "a_thinking_model_returned_64_empty_cells_first": "qwen3.6 is a reasoning model: with thinking enabled every generated token went to the `thinking` field and `response` came back EMPTY (verified at num_predict 12 AND 256, both done_reason=length). The first run wrote 64 empty cells. Scored naively that is 0% on all three arms and a delta of exactly 0.000 — a publishable-looking null produced by a formatting bug. Fixed with think=False and an abort guard that stops the run if the first 8 cells are all empty.",
    "reproduce": "build_items.py (asserts no answer-label leakage, no marker in the redacted arm, parallel shapes, balanced answers) -> run_panel.py (append-only, resumable, temperature 0) -> analyse.py (gate, then per-member, then floor, then delta; refuses to emit a value on a failed gate). Public domain."
}