统计口径与修订

Score-level statistical fixes (stats_fixes.md)

Generated by stats_fixes.py from ../../data/report.json (report.json, 12-arm unified accounting). stdlib-only recomputation; exact binomial McNemar, Holm and Benjamini-Hochberg implemented by hand.

Conventions. Pool = union of task ids over the arms of that pool (Pro 731, Verified 500) and is the denominator of every rate and delta; a task missing from an arm counts as unsolved. For "A vs B" (or "A - B"): b = solved by A only, c = solved by B only, delta = (b - c)/N, SE = sqrt((b+c) - (b-c)^2/N)/N, Wald 95% CI = delta +/- 1.96 SE, p = two-sided exact binomial on the discordant pairs (min(1, 2 P[X <= min(b,c)]), X ~ Bin(b+c, 1/2)). MDE_80 = 2.8 SE. "Survives" = adjusted p <= 0.05. The Verified temperature-0.6 arms have no rows, so the 21 report.json tests involving them are void and excluded everywhere (132 - 21 = 111).

Arms: n27 = Qwen3.6-27B native; s128 = n27 + v12 corpus, 1 epoch; s256 = n27 + v12 corpus, 2 epochs; s381 = n27 + v12 corpus, 3 epochs; v13 = n27 + v13 corpus; q38 = Qwen3.8-27B native (vendor next generation); a3b_base = Qwen3.5-35B-A3B-Base (bare base); a3b_2ep = a3b_base + v12 corpus, 2 epochs; a3b_q36 = Qwen3.6-35B-A3B native; a3b_v12 = a3b_q36 + v12 corpus, 3 epochs; a3b_q36t06 = a3b_q36 weights sampled at temperature 0.6 (Pro only); a3b_v12t06 = a3b_v12 weights sampled at temperature 0.6 (Pro only)

0. Arm ledger (solved / pool N; missing = unsolved)

a. Exact McNemar for every same-pool pair, checked against report.json

111 pairs recomputed (Pro 66, Verified 45); 111 match report.json exactly in b, c, delta, CI and p (tolerance 1e-9); mismatches: none. Void report entries excluded: 21.

pro (N = 731)

verified (N = 500)

b. Multiplicity (Holm = FWER, BH = FDR; alpha 0.05)

Family (ii): 13 pre-declared baseline-relative corpus tests (m = 13)

delta = corpus arm - its own baseline (positive = the corpus helped).

• survive raw 0.05: pro|a3b_2ep - a3b_base
• survive Holm 0.05: pro|a3b_2ep - a3b_base
• survive BH 0.05: pro|a3b_2ep - a3b_base

Family (iii): 6 generation / architecture tests (m = 6)

delta = first-named - second-named: q38 - n27 (vendor generation jump), n27 - a3b_q36 (dense - sparse, native), s381 - a3b_v12 (dense - sparse, after the same corpus).

• survive raw 0.05: pro|q38 - n27, verified|q38 - n27, pro|n27 - a3b_q36, verified|n27 - a3b_q36, verified|s381 - a3b_v12
• survive Holm 0.05: pro|q38 - n27, verified|q38 - n27, pro|n27 - a3b_q36, verified|n27 - a3b_q36, verified|s381 - a3b_v12
• survive BH 0.05: pro|q38 - n27, verified|q38 - n27, pro|n27 - a3b_q36, verified|n27 - a3b_q36, verified|s381 - a3b_v12

Family (i): all 111 valid same-pool pairs

• survive raw 0.05: 74 tests
• survive Holm 0.05: 49 tests
• survive BH 0.05: 71 tests

Tests that are raw-significant but do NOT survive Holm (Pro then Verified):

Full 111-row table with adjusted p:

c. Minimum detectable effect

c1. Paired design, from the observed discordance (k = 1 rollout): SE = sqrt((b+c) - (b-c)^2/N)/N, MDE_80 = 2.8 SE

pro (N = 731) — median SE 1.44 pp, median MDE_80 4.03 pp (median discordant pairs b+c = 111); over all 66 pro pairs the median MDE_80 is 4.07 pp.

N needed for MDE_80 = 3 pp at k = 1 from the observed discordance ((2.8/0.03)^2 x ((b+c)/N - delta^2)): median 1322 tasks.

verified (N = 500) — median SE 1.80 pp, median MDE_80 5.04 pp (median discordant pairs b+c = 81.0); over all 45 verified pairs the median MDE_80 is 5.36 pp.

N needed for MDE_80 = 3 pp at k = 1 from the observed discordance ((2.8/0.03)^2 x ((b+c)/N - delta^2)): median 1412 tasks.

c2. Variance-decomposition MDE (rollout x k)

Inputs: sigma2_between_d = 0.0647, sigma2_within = 0.1322 (arm A) and 0.1378 (arm B); all three from the 237-task stratified rollout x3 subsample, not from the 731-task pool. Model: Var(delta_hat) = [sigma2_between_d + (sigma2_within_A + sigma2_within_B)/k] / N ; MDE_80 = 2.8*sqrt(Var) with sigma2_within_A + sigma2_within_B = 0.2700 (this is the reading that reproduces the narrative's "~4.1 pp at k = 3, N = 731"; treating the two values as alternative estimates of one pooled within-variance would give the smaller numbers listed below as a sensitivity line).

• N needed for MDE_80 = 3 pp: k = 1: N = 2916, k = 3: N = 1348, k = 8: N = 858, k = inf: N = 564.
• The earlier "4.6 pp floor" is 4.63 pp = 2.8 sqrt(0.0647/237): the k -> infinity value at N = 237, i.e. the size of the rollout x3 subsample, not of the evaluation pool. At N = 731 the same between-task variance gives 2.63 pp (k -> inf) and 4.07 pp (k = 3).
• Sensitivity (alternative reading, not used): within = 0.1322, N = 731, k = 1: 4.60 pp; within = 0.1322, N = 731, k = 3: 3.42 pp; within = 0.1378, N = 731, k = 1: 4.66 pp; within = 0.1378, N = 731, k = 3: 3.44 pp.
• Consistency caveat: at k = 1 the decomposition implies a single-rollout discordance rate of about 0.335 (sigma2_between_d + within_A + within_B), whereas the observed (b+c)/N in the 731/500-task pools is 0.133-0.226 (baseline-relative pairs) and 0.114-0.133 for the same-weights temperature twins. The subsample therefore over-states the pool's run-to-run noise by roughly 2x in variance; the k = 1 row of c2 is an upper bound and the c1 numbers are the ones to quote for the actual design.

d. Temperature twins on Pro (same weights, t = 0.8 vs t = 0.6; delta = t0.6 - t0.8)

a3b_q36t06 vs a3b_q36: flips by language

a3b_q36t06 vs a3b_q36: flips by repo

a3b_v12t06 vs a3b_v12: flips by language

a3b_v12t06 vs a3b_v12: flips by repo

e. Decontamination slice: Pro without the teleport repo

repo values in the Pro rows: ansible (96), openlibrary (91), flipt (85), qutebrowser (79), teleport (76), webclients (65), vuls (62), navidrome (57), element-web (56), NodeBB (44), tutanota (20). The teleport repo is teleport: 76 tasks; Pro minus teleport = 655 tasks. (Verified contains no teleport tasks, so its tests are unchanged.)

e1. Per-arm solved counts and rates (denominator = subset size; the "rows" columns use only the tasks the arm actually ran, as the narrative's 37.3 -> 47.4 % figures do)

e2. The 7 Pro baseline-relative tests on Pro minus teleport (N = 655), Holm within these 7 (m = 7)

delta = (b - c)/655.

• survive raw 0.05: pro-minus-teleport|a3b_2ep - a3b_base
• survive Holm 0.05: pro-minus-teleport|a3b_2ep - a3b_base
• survive BH 0.05: pro-minus-teleport|a3b_2ep - a3b_base

e3. The 13-test family with the 7 Pro tests replaced by their teleport-excluded versions (Verified unchanged), Holm within 13 (m = 13)

• survive raw 0.05: pro-minus-teleport|a3b_2ep - a3b_base
• survive Holm 0.05: pro-minus-teleport|a3b_2ep - a3b_base
• survive BH 0.05: pro-minus-teleport|a3b_2ep - a3b_base

e4. Same 7 tests on teleport only (N = 76), for reference (m = 7)

delta = (b - c)/76.

• survive raw 0.05: pro-teleport|s381 - n27
• survive Holm 0.05: none
• survive BH 0.05: none

e5. Per-repo table, Pro (n tasks; solved per arm, missing = unsolved)

f. What changes for the paper (NARRATIVE_REPORT.md section H2 / H4)

• Nothing in report.json is wrong: all 111 valid pairs reproduce b, c, delta, the Wald CI (z = 1.96) and the exact McNemar p to 1e-9, so every raw number quoted in H2/H4 stands; what changes is the wording around corrections, the MDE, and the decontamination slice.
• H2, pre-declared corpus family (13 tests): exactly one test survives Holm and BH at 0.05 — pro|a3b_2ep - a3b_base (+8.89 pp, p = 4.51e-07, Holm p = 5.87e-06). Every post-trained-base comparison is null after correction: s256 - n27 Pro +2.74 pp has exact p = 0.054, Holm-13 p = 0.653, BH q = 0.269; a3b_v12 - a3b_q36 Pro +2.74 pp has p = 0.075, Holm-13 p = 0.822. The sentence 's256 vs n27 on Pro is p = 0.054 with a Wald CI that just excludes zero' must not be read as evidence: the CI [+0.09, +5.38] excludes zero only through the normal approximation while the exact test does not reject; report the exact p as the decision and drop or re-label the Wald CI where the two disagree (same situation for a3b_v12 - a3b_q36 on Pro-minus-teleport: CI [+0.10, +6.01], p = 0.054).
• H2, the two 'exceptions' (v13 vs s381 Verified -5.00 pp, p = 0.005; s256 vs s128 Pro +3.15 pp, p = 0.014) do not survive Holm over the 111 valid tests (Holm p = 0.250 and 0.658); they do survive BH (q = 0.009 and 0.024). Neither is a pre-declared endpoint (both compare checkpoints with each other), so word them as exploratory, 'significant before correction, retained at FDR 0.05, not at FWER 0.05'. The v13-mask cost of -5.0 pp on Verified (H4, reconciliations) carries the same qualifier.
• H2, generation jump: q38 - n27 survives Holm within the 6-test generation/architecture family on both pools (Pro +5.61 pp, Holm p = 0.000537; Verified +5.80 pp, Holm p = 0.009). Over the 111-test family only the Pro test survives Holm (adj. p = 0.007); Verified has Holm-111 p = 0.127 but BH-111 q = 0.005. Say which family the correction is over; recommended wording: 'significant after Holm within the pre-declared family on both pools; after Holm over all 111 tests on Pro only'.
• H2, dense vs sparse: n27 - a3b_q36 survives Holm within the 6-test family on both pools (Pro +3.97 pp, Holm p = 0.023; Verified +5.40 pp, Holm p = 0.023) but not Holm-111 (adj. p = 0.387 / 0.467; BH-111 q = 0.014 / 0.017). After the same corpus, s381 - a3b_v12 is +2.19 pp (p = 0.137) on Pro — null under every correction — and +8.20 pp on Verified, which survives everything (Holm-111 p = 0.000974). 'Score follows the architecture' should therefore be stated as: same sign on both pools, established on Verified, not resolved on Pro.
• H2, 'Holm/FDR over the 111 paired tests' (the announced fix): 74 of 111 tests are raw-significant, 49 survive Holm, 71 survive BH; the 25 tests that Holm removes are all |delta| <= 5.8 pp (raw p >= 0.0011). Suggested policy for the paper: pre-declared families with Holm as the primary decision rule (13 corpus tests; 6 generation/architecture tests), Holm over all 111 as a sensitivity, BH only as a screening statement; one p-value per test.
• H2, decontamination / teleport (repo value teleport, 76/731 tasks; Pro-minus-teleport = 655): the narrative's 's381 gain sits entirely in teleport (37.3 -> 47.4 % vs 52.0 -> 51.8 % elsewhere)' is confirmed with per-arm-rows denominators; with the pool convention it reads 36.8 -> 47.4 % on teleport vs 51.6 -> 51.5 % elsewhere. On Pro-minus-teleport (N = 655): s128 -1.22 (p = 0.456), s256 +2.60 [-0.19, +5.38] (p = 0.086), s381 -0.15 (p = 1.000), v13 +1.98 (p = 0.218) vs n27; a3b_v12 - a3b_q36 +3.05 (p = 0.054); a3b_2ep - a3b_base +9.62 [+6.09, +13.15] (p = 1.7e-07); temperature-0.6 pair +1.83 (p = 0.294). Holm within these 7 (and within the 13 with Verified unchanged) again leaves only a3b_2ep - a3b_base. Decontamination statement to write: excluding teleport changes no decision (nothing becomes significant, nothing significant becomes null); it removes s381's apparent Pro gain (+0.96 -> -0.15 pp) and slightly strengthens s256 (+2.74 -> +2.60) and a3b_v12 (+2.74 -> +3.05), i.e. the post-trained-base corpus effect is not a teleport artefact — it is simply below the MDE everywhere.
• H2, teleport as a repo-level (not instance-level) advantage: on the 76 teleport tasks every v12/v13 checkpoint beats n27 — s128 +6.58, s256 +3.95, s381 +10.53 (b = 8, c = 0, p = 0.008, Holm-7 p = 0.055), v13 +3.95 pp — while elsewhere the same checkpoints are -1.2 to +2.6 pp. This is consistent with the corpus's 1,622 teleport trajectories helping on the repo even though instance-level matches are zero; the paper should say so and keep the Pro-minus-teleport table as the primary Pro table for corpus effects. Caveat for the same paragraph: a 76-task slice has SE about 4 pp and the same-weights twins flip 9/9 teleport tasks between temperatures, so 'entirely in teleport' is a one-rollout observation, not a measured effect.
• H4, run-to-run: the quoted flips are exact — 97/731 = 13.3 % (a3b_q36) and 83/731 = 11.4 % (a3b_v12), net +0.96 pp (p = 0.543) and +0.68 pp (p = 0.661); restricting to tasks present in both runs gives the same rates (97/730, 82/722), so the missing-task convention is not driving them. New material to add: flip rate by language ranges from Go 10.4 % (n = 280) / Go 8.6 % (n = 280) to JavaScript 25.0 % (n = 44) / JavaScript 20.5 % (n = 44); by repo from flipt 4.7 % (n = 85) / flipt 1.2 % (n = 85) to NodeBB 25.0 % (n = 44) / NodeBB 20.5 % (n = 44) — the noise is not uniform across strata, which matters for any per-language claim.
• H4, MDE — replace the 3.8 pp / 4.6 pp sentences: (1) the '4.6 pp floor' is 4.63 pp = 2.8 sqrt(0.0647/237), the k -> infinity value at N = 237 (the rollout x3 subsample size); at N = 731 the same between-task variance gives 2.63 pp (k -> inf), 3.25 pp (k = 8), 4.07 pp (k = 3) and 5.99 pp (k = 1); at N = 500: 3.19 / 4.93 / 7.24 pp (k = inf / 3 / 1). (2) The '3.8 pp' figure is one pair (s256 - n27 on Pro); the median one-rollout paired MDE_80 over the pre-declared baseline-relative pairs is 4.03 pp on Pro (range 3.77-4.83) and 5.04 pp on Verified (range 4.87-5.94); quote these. (3) N for MDE_80 = 3 pp: from the observed one-rollout discordance about 1322 Pro tasks (k = 1); from the decomposition 1348 tasks at k = 3 and 564 at k -> inf. (4) State the provenance: the decomposition numbers are from the 237-task stratified rollout x3 subsample, and its implied single-rollout discordance (0.33) is about twice what the 731/500-task pools show (0.13-0.23; twins 0.11-0.13), so the k = 1 decomposition row is an upper bound; use the observed-discordance MDE (c1) for the actual design and the decomposition only for the k > 1 projections, labelled conservative.
• H4 accepted wording ('one-rollout paired pass@1 does not resolve most observed 0-3 pp SFT differences') is consistent with the corrected MDE (about 4 pp Pro, 5 pp Verified) and can stay; the 'Fixes' bullet of H2 (Holm/FDR, pre-declared endpoints, decontamination table) is now delivered by stats_fixes.md sections b, c and e and should be cited from there.