# Score-level statistical fixes (stats_fixes.md)

Generated by `stats_fixes.py` from `../../data/report.json` (report.json, 12-arm unified accounting). stdlib-only recomputation; exact binomial McNemar, Holm and Benjamini-Hochberg implemented by hand.

**Conventions.** Pool = union of task ids over the arms of that pool (Pro 731, Verified 500) and is the denominator of every rate and delta; a task missing from an arm counts as unsolved. For "A vs B" (or "A - B"): b = solved by A only, c = solved by B only, delta = (b - c)/N, SE = sqrt((b+c) - (b-c)^2/N)/N, Wald 95% CI = delta +/- 1.96 SE, p = two-sided exact binomial on the discordant pairs (min(1, 2 P[X <= min(b,c)]), X ~ Bin(b+c, 1/2)). MDE_80 = 2.8 SE. "Survives" = adjusted p <= 0.05. The Verified temperature-0.6 arms have no rows, so the 21 report.json tests involving them are void and excluded everywhere (132 - 21 = 111).

Arms: `n27` = Qwen3.6-27B native; `s128` = n27 + v12 corpus, 1 epoch; `s256` = n27 + v12 corpus, 2 epochs; `s381` = n27 + v12 corpus, 3 epochs; `v13` = n27 + v13 corpus; `q38` = Qwen3.8-27B native (vendor next generation); `a3b_base` = Qwen3.5-35B-A3B-Base (bare base); `a3b_2ep` = a3b_base + v12 corpus, 2 epochs; `a3b_q36` = Qwen3.6-35B-A3B native; `a3b_v12` = a3b_q36 + v12 corpus, 3 epochs; `a3b_q36t06` = a3b_q36 weights sampled at temperature 0.6 (Pro only); `a3b_v12t06` = a3b_v12 weights sampled at temperature 0.6 (Pro only)

## 0. Arm ledger (solved / pool N; missing = unsolved)

| arm|pool | rows | missing | solved | rate % (pool N) | rate % (rows) | report solved | matches report |
|---|---|---|---|---|---|---|---|
| q38|pro | 730 | 1 | 407 | 55.7 | 55.8 | 407 | yes |
| n27|pro | 725 | 6 | 366 | 50.1 | 50.5 | 366 | yes |
| s128|pro | 730 | 1 | 363 | 49.7 | 49.7 | 363 | yes |
| s256|pro | 722 | 9 | 386 | 52.8 | 53.5 | 386 | yes |
| s381|pro | 727 | 4 | 373 | 51.0 | 51.3 | 373 | yes |
| v13|pro | 731 | 0 | 382 | 52.3 | 52.3 | 382 | yes |
| a3b_base|pro | 730 | 1 | 259 | 35.4 | 35.5 | 259 | yes |
| a3b_2ep|pro | 722 | 9 | 324 | 44.3 | 44.9 | 324 | yes |
| a3b_q36|pro | 731 | 0 | 337 | 46.1 | 46.1 | 337 | yes |
| a3b_q36t06|pro | 730 | 1 | 344 | 47.1 | 47.1 | 344 | yes |
| a3b_v12|pro | 731 | 0 | 357 | 48.8 | 48.8 | 357 | yes |
| a3b_v12t06|pro | 722 | 9 | 362 | 49.5 | 50.1 | 362 | yes |
| q38|verified | 495 | 5 | 371 | 74.2 | 74.9 | 371 | yes |
| n27|verified | 496 | 4 | 342 | 68.4 | 69.0 | 342 | yes |
| s128|verified | 493 | 7 | 339 | 67.8 | 68.8 | 339 | yes |
| s256|verified | 494 | 6 | 343 | 68.6 | 69.4 | 343 | yes |
| s381|verified | 486 | 14 | 356 | 71.2 | 73.3 | 356 | yes |
| v13|verified | 496 | 4 | 331 | 66.2 | 66.7 | 331 | yes |
| a3b_base|verified | 497 | 3 | 260 | 52.0 | 52.3 | 260 | yes |
| a3b_2ep|verified | 493 | 7 | 277 | 55.4 | 56.2 | 277 | yes |
| a3b_q36|verified | 499 | 1 | 315 | 63.0 | 63.1 | 315 | yes |
| a3b_v12|verified | 496 | 4 | 315 | 63.0 | 63.5 | 315 | yes |

## a. Exact McNemar for every same-pool pair, checked against report.json

111 pairs recomputed (Pro 66, Verified 45); 111 match report.json exactly in b, c, delta, CI and p (tolerance 1e-9); mismatches: none. Void report entries excluded: 21.

### pro (N = 731)

| pair (A vs B) | b | c | both | neither | delta pp | 95% CI pp | p exact | report p | match |
|---|---|---|---|---|---|---|---|---|---|
| pro|n27 vs q38 | 34 | 75 | 332 | 290 | -5.61 | [-8.38, -2.84] | 0.000107 | 0.000107 | yes |
| pro|s128 vs q38 | 34 | 78 | 329 | 290 | -6.02 | [-8.82, -3.22] | 3.91e-05 | 3.91e-05 | yes |
| pro|s128 vs n27 | 47 | 50 | 316 | 318 | -0.41 | [-3.05, +2.23] | 0.839 | 0.839 | yes |
| pro|s256 vs q38 | 41 | 62 | 345 | 283 | -2.87 | [-5.59, -0.16] | 0.048 | 0.048 | yes |
| pro|s256 vs n27 | 59 | 39 | 327 | 306 | +2.74 | [+0.09, +5.38] | 0.054 | 0.054 | yes |
| pro|s256 vs s128 | 52 | 29 | 334 | 316 | +3.15 | [+0.74, +5.55] | 0.014 | 0.014 | yes |
| pro|s381 vs q38 | 35 | 69 | 338 | 289 | -4.65 | [-7.36, -1.94] | 0.001 | 0.001 | yes |
| pro|s381 vs n27 | 59 | 52 | 314 | 306 | +0.96 | [-1.87, +3.78] | 0.569 | 0.569 | yes |
| pro|s381 vs s128 | 50 | 40 | 323 | 318 | +1.37 | [-1.17, +3.91] | 0.343 | 0.343 | yes |
| pro|s381 vs s256 | 32 | 45 | 341 | 313 | -1.78 | [-4.13, +0.57] | 0.171 | 0.171 | yes |
| pro|v13 vs q38 | 38 | 63 | 344 | 286 | -3.42 | [-6.10, -0.74] | 0.017 | 0.017 | yes |
| pro|v13 vs n27 | 61 | 45 | 321 | 304 | +2.19 | [-0.57, +4.94] | 0.145 | 0.145 | yes |
| pro|v13 vs s128 | 58 | 39 | 324 | 310 | +2.60 | [-0.03, +5.23] | 0.067 | 0.067 | yes |
| pro|v13 vs s256 | 39 | 43 | 343 | 306 | -0.55 | [-2.97, +1.88] | 0.741 | 0.741 | yes |
| pro|v13 vs s381 | 56 | 47 | 326 | 302 | +1.23 | [-1.49, +3.95] | 0.431 | 0.431 | yes |
| pro|a3b_base vs q38 | 22 | 170 | 237 | 302 | -20.25 | [-23.66, -16.83] | 1.59e-29 | 1.59e-29 | yes |
| pro|a3b_base vs n27 | 23 | 130 | 236 | 342 | -14.64 | [-17.78, -11.50] | 2.54e-19 | 2.54e-19 | yes |
| pro|a3b_base vs s128 | 35 | 139 | 224 | 333 | -14.23 | [-17.61, -10.84] | 7.17e-16 | 7.17e-16 | yes |
| pro|a3b_base vs s256 | 19 | 146 | 240 | 326 | -17.37 | [-20.58, -14.17] | 1.86e-25 | 1.86e-25 | yes |
| pro|a3b_base vs s381 | 35 | 149 | 224 | 323 | -15.60 | [-19.05, -12.14] | 6e-18 | 6e-18 | yes |
| pro|a3b_base vs v13 | 25 | 148 | 234 | 324 | -16.83 | [-20.14, -13.52] | 1.87e-22 | 1.87e-22 | yes |
| pro|a3b_2ep vs q38 | 29 | 112 | 295 | 295 | -11.35 | [-14.43, -8.28] | 1.04e-12 | 1.04e-12 | yes |
| pro|a3b_2ep vs n27 | 40 | 82 | 284 | 325 | -5.75 | [-8.68, -2.81] | 0.00018 | 0.00018 | yes |
| pro|a3b_2ep vs s128 | 38 | 77 | 286 | 330 | -5.34 | [-8.18, -2.49] | 0.000351 | 0.000351 | yes |
| pro|a3b_2ep vs s256 | 32 | 94 | 292 | 313 | -8.48 | [-11.43, -5.54] | 2.91e-08 | 2.91e-08 | yes |
| pro|a3b_2ep vs s381 | 40 | 89 | 284 | 318 | -6.70 | [-9.71, -3.70] | 1.91e-05 | 1.91e-05 | yes |
| pro|a3b_2ep vs v13 | 36 | 94 | 288 | 313 | -7.93 | [-10.94, -4.93] | 3.73e-07 | 3.73e-07 | yes |
| pro|a3b_2ep vs a3b_base | 115 | 50 | 209 | 357 | +8.89 | [+5.51, +12.28] | 4.51e-07 | 4.51e-07 | yes |
| pro|a3b_q36 vs q38 | 36 | 106 | 301 | 288 | -9.58 | [-12.69, -6.46] | 3.39e-09 | 3.39e-09 | yes |
| pro|a3b_q36 vs n27 | 41 | 70 | 296 | 324 | -3.97 | [-6.78, -1.16] | 0.008 | 0.008 | yes |
| pro|a3b_q36 vs s128 | 51 | 77 | 286 | 317 | -3.56 | [-6.58, -0.53] | 0.027 | 0.027 | yes |
| pro|a3b_q36 vs s256 | 40 | 89 | 297 | 305 | -6.70 | [-9.71, -3.70] | 1.91e-05 | 1.91e-05 | yes |
| pro|a3b_q36 vs s381 | 50 | 86 | 287 | 308 | -4.92 | [-8.03, -1.82] | 0.003 | 0.003 | yes |
| pro|a3b_q36 vs v13 | 42 | 87 | 295 | 307 | -6.16 | [-9.17, -3.14] | 9.21e-05 | 9.21e-05 | yes |
| pro|a3b_q36 vs a3b_base | 112 | 34 | 225 | 360 | +10.67 | [+7.52, +13.82] | 6.42e-11 | 6.42e-11 | yes |
| pro|a3b_q36 vs a3b_2ep | 77 | 64 | 260 | 330 | +1.78 | [-1.40, +4.96] | 0.312 | 0.312 | yes |
| pro|a3b_q36t06 vs q38 | 41 | 104 | 303 | 283 | -8.62 | [-11.79, -5.45] | 1.7e-07 | 1.7e-07 | yes |
| pro|a3b_q36t06 vs n27 | 41 | 63 | 303 | 324 | -3.01 | [-5.74, -0.28] | 0.039 | 0.039 | yes |
| pro|a3b_q36t06 vs s128 | 54 | 73 | 290 | 314 | -2.60 | [-5.61, +0.42] | 0.110 | 0.110 | yes |
| pro|a3b_q36t06 vs s256 | 49 | 91 | 295 | 296 | -5.75 | [-8.89, -2.60] | 0.000486 | 0.000486 | yes |
| pro|a3b_q36t06 vs s381 | 52 | 81 | 292 | 306 | -3.97 | [-7.05, -0.89] | 0.015 | 0.015 | yes |
| pro|a3b_q36t06 vs v13 | 49 | 87 | 295 | 300 | -5.20 | [-8.30, -2.09] | 0.001 | 0.001 | yes |
| pro|a3b_q36t06 vs a3b_base | 114 | 29 | 230 | 358 | +11.63 | [+8.53, +14.72] | 4.09e-13 | 4.09e-13 | yes |
| pro|a3b_q36t06 vs a3b_2ep | 83 | 63 | 261 | 324 | +2.74 | [-0.50, +5.97] | 0.116 | 0.116 | yes |
| pro|a3b_q36t06 vs a3b_q36 | 52 | 45 | 292 | 342 | +0.96 | [-1.68, +3.60] | 0.543 | 0.543 | yes |
| pro|a3b_v12 vs q38 | 30 | 80 | 327 | 294 | -6.84 | [-9.61, -4.07] | 2.02e-06 | 2.02e-06 | yes |
| pro|a3b_v12 vs n27 | 42 | 51 | 315 | 323 | -1.23 | [-3.82, +1.35] | 0.407 | 0.407 | yes |
| pro|a3b_v12 vs s128 | 44 | 50 | 313 | 324 | -0.82 | [-3.42, +1.78] | 0.606 | 0.606 | yes |
| pro|a3b_v12 vs s256 | 25 | 54 | 332 | 320 | -3.97 | [-6.33, -1.60] | 0.001 | 0.001 | yes |
| pro|a3b_v12 vs s381 | 43 | 59 | 314 | 315 | -2.19 | [-4.89, +0.51] | 0.137 | 0.137 | yes |
| pro|a3b_v12 vs v13 | 32 | 57 | 325 | 317 | -3.42 | [-5.94, -0.90] | 0.011 | 0.011 | yes |
| pro|a3b_v12 vs a3b_base | 122 | 24 | 235 | 350 | +13.41 | [+10.32, +16.50] | 5.33e-17 | 5.33e-17 | yes |
| pro|a3b_v12 vs a3b_2ep | 73 | 40 | 284 | 334 | +4.51 | [+1.68, +7.35] | 0.002 | 0.002 | yes |
| pro|a3b_v12 vs a3b_q36 | 67 | 47 | 290 | 327 | +2.74 | [-0.12, +5.59] | 0.075 | 0.075 | yes |
| pro|a3b_v12 vs a3b_q36t06 | 63 | 50 | 294 | 324 | +1.78 | [-1.07, +4.63] | 0.259 | 0.259 | yes |
| pro|a3b_v12t06 vs q38 | 37 | 82 | 325 | 287 | -6.16 | [-9.05, -3.27] | 4.53e-05 | 4.53e-05 | yes |
| pro|a3b_v12t06 vs n27 | 47 | 51 | 315 | 318 | -0.55 | [-3.20, +2.11] | 0.762 | 0.762 | yes |
| pro|a3b_v12t06 vs s128 | 44 | 45 | 318 | 324 | -0.14 | [-2.67, +2.39] | 1.000 | 1.000 | yes |
| pro|a3b_v12t06 vs s256 | 34 | 58 | 328 | 311 | -3.28 | [-5.84, -0.72] | 0.016 | 0.016 | yes |
| pro|a3b_v12t06 vs s381 | 45 | 56 | 317 | 313 | -1.50 | [-4.20, +1.19] | 0.320 | 0.320 | yes |
| pro|a3b_v12t06 vs v13 | 40 | 60 | 322 | 309 | -2.74 | [-5.41, -0.06] | 0.057 | 0.057 | yes |
| pro|a3b_v12t06 vs a3b_base | 135 | 32 | 227 | 337 | +14.09 | [+10.78, +17.40] | 2.96e-16 | 2.96e-16 | yes |
| pro|a3b_v12t06 vs a3b_2ep | 78 | 40 | 284 | 329 | +5.20 | [+2.31, +8.09] | 0.000598 | 0.000598 | yes |
| pro|a3b_v12t06 vs a3b_q36 | 68 | 43 | 294 | 326 | +3.42 | [+0.61, +6.23] | 0.022 | 0.022 | yes |
| pro|a3b_v12t06 vs a3b_q36t06 | 70 | 52 | 292 | 317 | +2.46 | [-0.49, +5.42] | 0.123 | 0.123 | yes |
| pro|a3b_v12t06 vs a3b_v12 | 44 | 39 | 318 | 330 | +0.68 | [-1.76, +3.13] | 0.661 | 0.661 | yes |

### verified (N = 500)

| pair (A vs B) | b | c | both | neither | delta pp | 95% CI pp | p exact | report p | match |
|---|---|---|---|---|---|---|---|---|---|
| verified|n27 vs q38 | 28 | 57 | 314 | 101 | -5.80 | [-9.38, -2.22] | 0.002 | 0.002 | yes |
| verified|s128 vs q38 | 17 | 49 | 322 | 112 | -6.40 | [-9.53, -3.27] | 0.000101 | 0.000101 | yes |
| verified|s128 vs n27 | 38 | 41 | 301 | 120 | -0.60 | [-4.08, +2.88] | 0.822 | 0.822 | yes |
| verified|s256 vs q38 | 22 | 50 | 321 | 107 | -5.60 | [-8.89, -2.31] | 0.001 | 0.001 | yes |
| verified|s256 vs n27 | 42 | 41 | 301 | 116 | +0.20 | [-3.37, +3.77] | 1.000 | 1.000 | yes |
| verified|s256 vs s128 | 35 | 31 | 308 | 126 | +0.80 | [-2.38, +3.98] | 0.712 | 0.712 | yes |
| verified|s381 vs q38 | 30 | 45 | 326 | 99 | -3.00 | [-6.38, +0.38] | 0.105 | 0.105 | yes |
| verified|s381 vs n27 | 45 | 31 | 311 | 113 | +2.80 | [-0.61, +6.21] | 0.135 | 0.135 | yes |
| verified|s381 vs s128 | 40 | 23 | 316 | 121 | +3.40 | [+0.30, +6.50] | 0.043 | 0.043 | yes |
| verified|s381 vs s256 | 36 | 23 | 320 | 121 | +2.60 | [-0.40, +5.60] | 0.117 | 0.117 | yes |
| verified|v13 vs q38 | 16 | 56 | 315 | 113 | -8.00 | [-11.25, -4.75] | 2.4e-06 | 2.4e-06 | yes |
| verified|v13 vs n27 | 34 | 45 | 297 | 124 | -2.20 | [-5.68, +1.28] | 0.260 | 0.260 | yes |
| verified|v13 vs s128 | 28 | 36 | 303 | 133 | -1.60 | [-4.73, +1.53] | 0.382 | 0.382 | yes |
| verified|v13 vs s256 | 25 | 37 | 306 | 132 | -2.40 | [-5.48, +0.68] | 0.162 | 0.162 | yes |
| verified|v13 vs s381 | 24 | 49 | 307 | 120 | -5.00 | [-8.32, -1.68] | 0.005 | 0.005 | yes |
| verified|a3b_base vs q38 | 17 | 128 | 243 | 112 | -22.20 | [-26.50, -17.90] | 3.03e-22 | 3.03e-22 | yes |
| verified|a3b_base vs n27 | 32 | 114 | 228 | 126 | -16.40 | [-20.91, -11.89] | 5.42e-12 | 5.42e-12 | yes |
| verified|a3b_base vs s128 | 28 | 107 | 232 | 133 | -15.80 | [-20.14, -11.46] | 4.44e-12 | 4.44e-12 | yes |
| verified|a3b_base vs s256 | 29 | 112 | 231 | 128 | -16.60 | [-21.02, -12.18] | 1.04e-12 | 1.04e-12 | yes |
| verified|a3b_base vs s381 | 22 | 118 | 238 | 122 | -19.20 | [-23.52, -14.88] | 4.49e-17 | 4.49e-17 | yes |
| verified|a3b_base vs v13 | 29 | 100 | 231 | 140 | -14.20 | [-18.47, -9.93] | 2.47e-10 | 2.47e-10 | yes |
| verified|a3b_2ep vs q38 | 13 | 107 | 264 | 116 | -18.80 | [-22.77, -14.83] | 1.5e-19 | 1.5e-19 | yes |
| verified|a3b_2ep vs n27 | 29 | 94 | 248 | 129 | -13.00 | [-17.20, -8.80] | 3.39e-09 | 3.39e-09 | yes |
| verified|a3b_2ep vs s128 | 25 | 87 | 252 | 136 | -12.40 | [-16.40, -8.40] | 3.23e-09 | 3.23e-09 | yes |
| verified|a3b_2ep vs s256 | 22 | 88 | 255 | 135 | -13.20 | [-17.15, -9.25] | 1.55e-10 | 1.55e-10 | yes |
| verified|a3b_2ep vs s381 | 14 | 93 | 263 | 130 | -15.80 | [-19.61, -11.99] | 1.76e-15 | 1.76e-15 | yes |
| verified|a3b_2ep vs v13 | 26 | 80 | 251 | 143 | -10.80 | [-14.72, -6.88] | 1.43e-07 | 1.43e-07 | yes |
| verified|a3b_2ep vs a3b_base | 65 | 48 | 212 | 175 | +3.40 | [-0.76, +7.56] | 0.132 | 0.132 | yes |
| verified|a3b_q36 vs q38 | 20 | 76 | 295 | 109 | -11.20 | [-14.91, -7.49] | 7.32e-09 | 7.32e-09 | yes |
| verified|a3b_q36 vs n27 | 37 | 64 | 278 | 121 | -5.40 | [-9.31, -1.49] | 0.009 | 0.009 | yes |
| verified|a3b_q36 vs s128 | 39 | 63 | 276 | 122 | -4.80 | [-8.74, -0.86] | 0.022 | 0.022 | yes |
| verified|a3b_q36 vs s256 | 34 | 62 | 281 | 123 | -5.60 | [-9.41, -1.79] | 0.006 | 0.006 | yes |
| verified|a3b_q36 vs s381 | 31 | 72 | 284 | 113 | -8.20 | [-12.11, -4.29] | 6.6e-05 | 6.6e-05 | yes |
| verified|a3b_q36 vs v13 | 38 | 54 | 277 | 131 | -3.20 | [-6.95, +0.55] | 0.117 | 0.117 | yes |
| verified|a3b_q36 vs a3b_base | 90 | 35 | 225 | 150 | +11.00 | [+6.72, +15.28] | 9.24e-07 | 9.24e-07 | yes |
| verified|a3b_q36 vs a3b_2ep | 75 | 37 | 240 | 148 | +7.60 | [+3.51, +11.69] | 0.000421 | 0.000421 | yes |
| verified|a3b_v12 vs q38 | 16 | 72 | 299 | 113 | -11.20 | [-14.74, -7.66] | 1.19e-09 | 1.19e-09 | yes |
| verified|a3b_v12 vs n27 | 38 | 65 | 277 | 120 | -5.40 | [-9.35, -1.45] | 0.010 | 0.010 | yes |
| verified|a3b_v12 vs s128 | 25 | 49 | 290 | 136 | -4.80 | [-8.15, -1.45] | 0.007 | 0.007 | yes |
| verified|a3b_v12 vs s256 | 27 | 55 | 288 | 130 | -5.60 | [-9.12, -2.08] | 0.003 | 0.003 | yes |
| verified|a3b_v12 vs s381 | 23 | 64 | 292 | 121 | -8.20 | [-11.78, -4.62] | 1.27e-05 | 1.27e-05 | yes |
| verified|a3b_v12 vs v13 | 35 | 51 | 280 | 134 | -3.20 | [-6.82, +0.42] | 0.105 | 0.105 | yes |
| verified|a3b_v12 vs a3b_base | 91 | 36 | 224 | 149 | +11.00 | [+6.69, +15.31] | 1.14e-06 | 1.14e-06 | yes |
| verified|a3b_v12 vs a3b_2ep | 67 | 29 | 248 | 156 | +7.60 | [+3.82, +11.38] | 0.000132 | 0.000132 | yes |
| verified|a3b_v12 vs a3b_q36 | 45 | 45 | 270 | 140 | +0.00 | [-3.72, +3.72] | 1.000 | 1.000 | yes |

## b. Multiplicity (Holm = FWER, BH = FDR; alpha 0.05)

### Family (ii): 13 pre-declared baseline-relative corpus tests (m = 13)

delta = corpus arm - its own baseline (positive = the corpus helped).

| test (A - B) | b (A only) | c (B only) | delta pp | 95% CI pp | p exact | p Holm | q BH | raw<.05 | Holm<.05 | BH<.05 |
|---|---|---|---|---|---|---|---|---|---|---|
| pro|s128 - n27 | 47 | 50 | -0.41 | [-3.05, +2.23] | 0.839 | 1.000 | 0.992 | no | no | no |
| pro|s256 - n27 | 59 | 39 | +2.74 | [+0.09, +5.38] | 0.054 | 0.653 | 0.269 | no | no | no |
| pro|s381 - n27 | 59 | 52 | +0.96 | [-1.87, +3.78] | 0.569 | 1.000 | 0.822 | no | no | no |
| pro|v13 - n27 | 61 | 45 | +2.19 | [-0.57, +4.94] | 0.145 | 1.000 | 0.269 | no | no | no |
| verified|s128 - n27 | 38 | 41 | -0.60 | [-4.08, +2.88] | 0.822 | 1.000 | 0.992 | no | no | no |
| verified|s256 - n27 | 42 | 41 | +0.20 | [-3.37, +3.77] | 1.000 | 1.000 | 1.000 | no | no | no |
| verified|s381 - n27 | 45 | 31 | +2.80 | [-0.61, +6.21] | 0.135 | 1.000 | 0.269 | no | no | no |
| verified|v13 - n27 | 34 | 45 | -2.20 | [-5.68, +1.28] | 0.260 | 1.000 | 0.423 | no | no | no |
| pro|a3b_2ep - a3b_base | 115 | 50 | +8.89 | [+5.51, +12.28] | 4.51e-07 | 5.87e-06 | 5.87e-06 | yes | yes | yes |
| verified|a3b_2ep - a3b_base | 65 | 48 | +3.40 | [-0.76, +7.56] | 0.132 | 1.000 | 0.269 | no | no | no |
| pro|a3b_v12 - a3b_q36 | 67 | 47 | +2.74 | [-0.12, +5.59] | 0.075 | 0.822 | 0.269 | no | no | no |
| verified|a3b_v12 - a3b_q36 | 45 | 45 | +0.00 | [-3.72, +3.72] | 1.000 | 1.000 | 1.000 | no | no | no |
| pro|a3b_v12t06 - a3b_q36t06 | 70 | 52 | +2.46 | [-0.49, +5.42] | 0.123 | 1.000 | 0.269 | no | no | no |

- survive raw 0.05: pro|a3b_2ep - a3b_base
- survive Holm 0.05: pro|a3b_2ep - a3b_base
- survive BH 0.05: pro|a3b_2ep - a3b_base

### Family (iii): 6 generation / architecture tests (m = 6)

delta = first-named - second-named: q38 - n27 (vendor generation jump), n27 - a3b_q36 (dense - sparse, native), s381 - a3b_v12 (dense - sparse, after the same corpus).

| test (A - B) | b (A only) | c (B only) | delta pp | 95% CI pp | p exact | p Holm | q BH | raw<.05 | Holm<.05 | BH<.05 |
|---|---|---|---|---|---|---|---|---|---|---|
| pro|q38 - n27 | 75 | 34 | +5.61 | [+2.84, +8.38] | 0.000107 | 0.000537 | 0.000322 | yes | yes | yes |
| verified|q38 - n27 | 57 | 28 | +5.80 | [+2.22, +9.38] | 0.002 | 0.009 | 0.004 | yes | yes | yes |
| pro|n27 - a3b_q36 | 70 | 41 | +3.97 | [+1.16, +6.78] | 0.008 | 0.023 | 0.011 | yes | yes | yes |
| verified|n27 - a3b_q36 | 64 | 37 | +5.40 | [+1.49, +9.31] | 0.009 | 0.023 | 0.011 | yes | yes | yes |
| pro|s381 - a3b_v12 | 59 | 43 | +2.19 | [-0.51, +4.89] | 0.137 | 0.137 | 0.137 | no | no | no |
| verified|s381 - a3b_v12 | 64 | 23 | +8.20 | [+4.62, +11.78] | 1.27e-05 | 7.59e-05 | 7.59e-05 | yes | yes | yes |

- survive raw 0.05: pro|q38 - n27, verified|q38 - n27, pro|n27 - a3b_q36, verified|n27 - a3b_q36, verified|s381 - a3b_v12
- survive Holm 0.05: pro|q38 - n27, verified|q38 - n27, pro|n27 - a3b_q36, verified|n27 - a3b_q36, verified|s381 - a3b_v12
- survive BH 0.05: pro|q38 - n27, verified|q38 - n27, pro|n27 - a3b_q36, verified|n27 - a3b_q36, verified|s381 - a3b_v12

### Family (i): all 111 valid same-pool pairs

- survive raw 0.05: 74 tests
- survive Holm 0.05: 49 tests
- survive BH 0.05: 71 tests

Tests that are raw-significant but do NOT survive Holm (Pro then Verified):

| pair (A vs B) | delta pp | p exact | p Holm | q BH | BH<.05 |
|---|---|---|---|---|---|
| pro|s256 vs q38 | -2.87 | 0.048 | 1.000 | 0.072 | no |
| pro|s256 vs s128 | +3.15 | 0.014 | 0.658 | 0.024 | yes |
| pro|s381 vs q38 | -4.65 | 0.001 | 0.069 | 0.002 | yes |
| pro|v13 vs q38 | -3.42 | 0.017 | 0.726 | 0.027 | yes |
| pro|a3b_q36 vs n27 | -3.97 | 0.008 | 0.387 | 0.014 | yes |
| pro|a3b_q36 vs s128 | -3.56 | 0.027 | 1.000 | 0.042 | yes |
| pro|a3b_q36 vs s381 | -4.92 | 0.003 | 0.143 | 0.005 | yes |
| pro|a3b_q36t06 vs n27 | -3.01 | 0.039 | 1.000 | 0.060 | no |
| pro|a3b_q36t06 vs s381 | -3.97 | 0.015 | 0.684 | 0.025 | yes |
| pro|a3b_q36t06 vs v13 | -5.20 | 0.001 | 0.085 | 0.003 | yes |
| pro|a3b_v12 vs s256 | -3.97 | 0.001 | 0.086 | 0.003 | yes |
| pro|a3b_v12 vs v13 | -3.42 | 0.011 | 0.506 | 0.018 | yes |
| pro|a3b_v12 vs a3b_2ep | +4.51 | 0.002 | 0.140 | 0.005 | yes |
| pro|a3b_v12t06 vs s256 | -3.28 | 0.016 | 0.721 | 0.027 | yes |
| pro|a3b_v12t06 vs a3b_q36 | +3.42 | 0.022 | 0.959 | 0.035 | yes |
| verified|n27 vs q38 | -5.80 | 0.002 | 0.127 | 0.005 | yes |
| verified|s256 vs q38 | -5.60 | 0.001 | 0.079 | 0.003 | yes |
| verified|s381 vs s128 | +3.40 | 0.043 | 1.000 | 0.065 | no |
| verified|v13 vs s381 | -5.00 | 0.005 | 0.250 | 0.009 | yes |
| verified|a3b_q36 vs n27 | -5.40 | 0.009 | 0.467 | 0.017 | yes |
| verified|a3b_q36 vs s128 | -4.80 | 0.022 | 0.959 | 0.035 | yes |
| verified|a3b_q36 vs s256 | -5.60 | 0.006 | 0.295 | 0.010 | yes |
| verified|a3b_v12 vs n27 | -5.40 | 0.010 | 0.493 | 0.018 | yes |
| verified|a3b_v12 vs s128 | -4.80 | 0.007 | 0.368 | 0.013 | yes |
| verified|a3b_v12 vs s256 | -5.60 | 0.003 | 0.145 | 0.005 | yes |

Full 111-row table with adjusted p:

| pair (A vs B) | b | c | delta pp | p exact | p Holm | q BH | raw | Holm | BH |
|---|---|---|---|---|---|---|---|---|---|
| pro|n27 vs q38 | 34 | 75 | -5.61 | 0.000107 | 0.007 | 0.000277 | yes | yes | yes |
| pro|s128 vs q38 | 34 | 78 | -6.02 | 3.91e-05 | 0.003 | 0.000114 | yes | yes | yes |
| pro|s128 vs n27 | 47 | 50 | -0.41 | 0.839 | 1.000 | 0.863 | no | no | no |
| pro|s256 vs q38 | 41 | 62 | -2.87 | 0.048 | 1.000 | 0.072 | yes | no | no |
| pro|s256 vs n27 | 59 | 39 | +2.74 | 0.054 | 1.000 | 0.080 | no | no | no |
| pro|s256 vs s128 | 52 | 29 | +3.15 | 0.014 | 0.658 | 0.024 | yes | no | yes |
| pro|s381 vs q38 | 35 | 69 | -4.65 | 0.001 | 0.069 | 0.002 | yes | no | yes |
| pro|s381 vs n27 | 59 | 52 | +0.96 | 0.569 | 1.000 | 0.626 | no | no | no |
| pro|s381 vs s128 | 50 | 40 | +1.37 | 0.343 | 1.000 | 0.396 | no | no | no |
| pro|s381 vs s256 | 32 | 45 | -1.78 | 0.171 | 1.000 | 0.209 | no | no | no |
| pro|v13 vs q38 | 38 | 63 | -3.42 | 0.017 | 0.726 | 0.027 | yes | no | yes |
| pro|v13 vs n27 | 61 | 45 | +2.19 | 0.145 | 1.000 | 0.181 | no | no | no |
| pro|v13 vs s128 | 58 | 39 | +2.60 | 0.067 | 1.000 | 0.097 | no | no | no |
| pro|v13 vs s256 | 39 | 43 | -0.55 | 0.741 | 1.000 | 0.783 | no | no | no |
| pro|v13 vs s381 | 56 | 47 | +1.23 | 0.431 | 1.000 | 0.483 | no | no | no |
| pro|a3b_base vs q38 | 22 | 170 | -20.25 | 1.59e-29 | 1.76e-27 | 1.76e-27 | yes | yes | yes |
| pro|a3b_base vs n27 | 23 | 130 | -14.64 | 2.54e-19 | 2.7e-17 | 4.71e-18 | yes | yes | yes |
| pro|a3b_base vs s128 | 35 | 139 | -14.23 | 7.17e-16 | 7.25e-14 | 7.24e-15 | yes | yes | yes |
| pro|a3b_base vs s256 | 19 | 146 | -17.37 | 1.86e-25 | 2.05e-23 | 1.03e-23 | yes | yes | yes |
| pro|a3b_base vs s381 | 35 | 149 | -15.60 | 6e-18 | 6.31e-16 | 9.52e-17 | yes | yes | yes |
| pro|a3b_base vs v13 | 25 | 148 | -16.83 | 1.87e-22 | 2.03e-20 | 6.9e-21 | yes | yes | yes |
| pro|a3b_2ep vs q38 | 29 | 112 | -11.35 | 1.04e-12 | 1.02e-10 | 7.73e-12 | yes | yes | yes |
| pro|a3b_2ep vs n27 | 40 | 82 | -5.75 | 0.00018 | 0.012 | 0.000444 | yes | yes | yes |
| pro|a3b_2ep vs s128 | 38 | 77 | -5.34 | 0.000351 | 0.023 | 0.000847 | yes | yes | yes |
| pro|a3b_2ep vs s256 | 32 | 94 | -8.48 | 2.91e-08 | 2.5e-06 | 1.24e-07 | yes | yes | yes |
| pro|a3b_2ep vs s381 | 40 | 89 | -6.70 | 1.91e-05 | 0.001 | 5.74e-05 | yes | yes | yes |
| pro|a3b_2ep vs v13 | 36 | 94 | -7.93 | 3.73e-07 | 3.1e-05 | 1.43e-06 | yes | yes | yes |
| pro|a3b_2ep vs a3b_base | 115 | 50 | +8.89 | 4.51e-07 | 3.7e-05 | 1.67e-06 | yes | yes | yes |
| pro|a3b_q36 vs q38 | 36 | 106 | -9.58 | 3.39e-09 | 3.01e-07 | 1.57e-08 | yes | yes | yes |
| pro|a3b_q36 vs n27 | 41 | 70 | -3.97 | 0.008 | 0.387 | 0.014 | yes | no | yes |
| pro|a3b_q36 vs s128 | 51 | 77 | -3.56 | 0.027 | 1.000 | 0.042 | yes | no | yes |
| pro|a3b_q36 vs s256 | 40 | 89 | -6.70 | 1.91e-05 | 0.001 | 5.74e-05 | yes | yes | yes |
| pro|a3b_q36 vs s381 | 50 | 86 | -4.92 | 0.003 | 0.143 | 0.005 | yes | no | yes |
| pro|a3b_q36 vs v13 | 42 | 87 | -6.16 | 9.21e-05 | 0.007 | 0.000249 | yes | yes | yes |
| pro|a3b_q36 vs a3b_base | 112 | 34 | +10.67 | 6.42e-11 | 6.04e-09 | 3.96e-10 | yes | yes | yes |
| pro|a3b_q36 vs a3b_2ep | 77 | 64 | +1.78 | 0.312 | 1.000 | 0.369 | no | no | no |
| pro|a3b_q36t06 vs q38 | 41 | 104 | -8.62 | 1.7e-07 | 1.43e-05 | 6.73e-07 | yes | yes | yes |
| pro|a3b_q36t06 vs n27 | 41 | 63 | -3.01 | 0.039 | 1.000 | 0.060 | yes | no | no |
| pro|a3b_q36t06 vs s128 | 54 | 73 | -2.60 | 0.110 | 1.000 | 0.151 | no | no | no |
| pro|a3b_q36t06 vs s256 | 49 | 91 | -5.75 | 0.000486 | 0.031 | 0.001 | yes | yes | yes |
| pro|a3b_q36t06 vs s381 | 52 | 81 | -3.97 | 0.015 | 0.684 | 0.025 | yes | no | yes |
| pro|a3b_q36t06 vs v13 | 49 | 87 | -5.20 | 0.001 | 0.085 | 0.003 | yes | no | yes |
| pro|a3b_q36t06 vs a3b_base | 114 | 29 | +11.63 | 4.09e-13 | 4.05e-11 | 3.49e-12 | yes | yes | yes |
| pro|a3b_q36t06 vs a3b_2ep | 83 | 63 | +2.74 | 0.116 | 1.000 | 0.155 | no | no | no |
| pro|a3b_q36t06 vs a3b_q36 | 52 | 45 | +0.96 | 0.543 | 1.000 | 0.602 | no | no | no |
| pro|a3b_v12 vs q38 | 30 | 80 | -6.84 | 2.02e-06 | 0.000159 | 6.79e-06 | yes | yes | yes |
| pro|a3b_v12 vs n27 | 42 | 51 | -1.23 | 0.407 | 1.000 | 0.461 | no | no | no |
| pro|a3b_v12 vs s128 | 44 | 50 | -0.82 | 0.606 | 1.000 | 0.660 | no | no | no |
| pro|a3b_v12 vs s256 | 25 | 54 | -3.97 | 0.001 | 0.086 | 0.003 | yes | no | yes |
| pro|a3b_v12 vs s381 | 43 | 59 | -2.19 | 0.137 | 1.000 | 0.173 | no | no | no |
| pro|a3b_v12 vs v13 | 32 | 57 | -3.42 | 0.011 | 0.506 | 0.018 | yes | no | yes |
| pro|a3b_v12 vs a3b_base | 122 | 24 | +13.41 | 5.33e-17 | 5.49e-15 | 6.57e-16 | yes | yes | yes |
| pro|a3b_v12 vs a3b_2ep | 73 | 40 | +4.51 | 0.002 | 0.140 | 0.005 | yes | no | yes |
| pro|a3b_v12 vs a3b_q36 | 67 | 47 | +2.74 | 0.075 | 1.000 | 0.106 | no | no | no |
| pro|a3b_v12 vs a3b_q36t06 | 63 | 50 | +1.78 | 0.259 | 1.000 | 0.311 | no | no | no |
| pro|a3b_v12t06 vs q38 | 37 | 82 | -6.16 | 4.53e-05 | 0.003 | 0.000129 | yes | yes | yes |
| pro|a3b_v12t06 vs n27 | 47 | 51 | -0.55 | 0.762 | 1.000 | 0.798 | no | no | no |
| pro|a3b_v12t06 vs s128 | 44 | 45 | -0.14 | 1.000 | 1.000 | 1.000 | no | no | no |
| pro|a3b_v12t06 vs s256 | 34 | 58 | -3.28 | 0.016 | 0.721 | 0.027 | yes | no | yes |
| pro|a3b_v12t06 vs s381 | 45 | 56 | -1.50 | 0.320 | 1.000 | 0.374 | no | no | no |
| pro|a3b_v12t06 vs v13 | 40 | 60 | -2.74 | 0.057 | 1.000 | 0.083 | no | no | no |
| pro|a3b_v12t06 vs a3b_base | 135 | 32 | +14.09 | 2.96e-16 | 3.02e-14 | 3.28e-15 | yes | yes | yes |
| pro|a3b_v12t06 vs a3b_2ep | 78 | 40 | +5.20 | 0.000598 | 0.038 | 0.001 | yes | yes | yes |
| pro|a3b_v12t06 vs a3b_q36 | 68 | 43 | +3.42 | 0.022 | 0.959 | 0.035 | yes | no | yes |
| pro|a3b_v12t06 vs a3b_q36t06 | 70 | 52 | +2.46 | 0.123 | 1.000 | 0.161 | no | no | no |
| pro|a3b_v12t06 vs a3b_v12 | 44 | 39 | +0.68 | 0.661 | 1.000 | 0.712 | no | no | no |
| verified|n27 vs q38 | 28 | 57 | -5.80 | 0.002 | 0.127 | 0.005 | yes | no | yes |
| verified|s128 vs q38 | 17 | 49 | -6.40 | 0.000101 | 0.007 | 0.000268 | yes | yes | yes |
| verified|s128 vs n27 | 38 | 41 | -0.60 | 0.822 | 1.000 | 0.853 | no | no | no |
| verified|s256 vs q38 | 22 | 50 | -5.60 | 0.001 | 0.079 | 0.003 | yes | no | yes |
| verified|s256 vs n27 | 42 | 41 | +0.20 | 1.000 | 1.000 | 1.000 | no | no | no |
| verified|s256 vs s128 | 35 | 31 | +0.80 | 0.712 | 1.000 | 0.760 | no | no | no |
| verified|s381 vs q38 | 30 | 45 | -3.00 | 0.105 | 1.000 | 0.146 | no | no | no |
| verified|s381 vs n27 | 45 | 31 | +2.80 | 0.135 | 1.000 | 0.173 | no | no | no |
| verified|s381 vs s128 | 40 | 23 | +3.40 | 0.043 | 1.000 | 0.065 | yes | no | no |
| verified|s381 vs s256 | 36 | 23 | +2.60 | 0.117 | 1.000 | 0.155 | no | no | no |
| verified|v13 vs q38 | 16 | 56 | -8.00 | 2.4e-06 | 0.000187 | 7.83e-06 | yes | yes | yes |
| verified|v13 vs n27 | 34 | 45 | -2.20 | 0.260 | 1.000 | 0.311 | no | no | no |
| verified|v13 vs s128 | 28 | 36 | -1.60 | 0.382 | 1.000 | 0.437 | no | no | no |
| verified|v13 vs s256 | 25 | 37 | -2.40 | 0.162 | 1.000 | 0.200 | no | no | no |
| verified|v13 vs s381 | 24 | 49 | -5.00 | 0.005 | 0.250 | 0.009 | yes | no | yes |
| verified|a3b_base vs q38 | 17 | 128 | -22.20 | 3.03e-22 | 3.27e-20 | 8.4e-21 | yes | yes | yes |
| verified|a3b_base vs n27 | 32 | 114 | -16.40 | 5.42e-12 | 5.15e-10 | 3.54e-11 | yes | yes | yes |
| verified|a3b_base vs s128 | 28 | 107 | -15.80 | 4.44e-12 | 4.26e-10 | 3.08e-11 | yes | yes | yes |
| verified|a3b_base vs s256 | 29 | 112 | -16.60 | 1.04e-12 | 1.02e-10 | 7.73e-12 | yes | yes | yes |
| verified|a3b_base vs s381 | 22 | 118 | -19.20 | 4.49e-17 | 4.67e-15 | 6.23e-16 | yes | yes | yes |
| verified|a3b_base vs v13 | 29 | 100 | -14.20 | 2.47e-10 | 2.27e-08 | 1.37e-09 | yes | yes | yes |
| verified|a3b_2ep vs q38 | 13 | 107 | -18.80 | 1.5e-19 | 1.6e-17 | 3.32e-18 | yes | yes | yes |
| verified|a3b_2ep vs n27 | 29 | 94 | -13.00 | 3.39e-09 | 3.01e-07 | 1.57e-08 | yes | yes | yes |
| verified|a3b_2ep vs s128 | 25 | 87 | -12.40 | 3.23e-09 | 2.9e-07 | 1.57e-08 | yes | yes | yes |
| verified|a3b_2ep vs s256 | 22 | 88 | -13.20 | 1.55e-10 | 1.44e-08 | 9.06e-10 | yes | yes | yes |
| verified|a3b_2ep vs s381 | 14 | 93 | -15.80 | 1.76e-15 | 1.76e-13 | 1.63e-14 | yes | yes | yes |
| verified|a3b_2ep vs v13 | 26 | 80 | -10.80 | 1.43e-07 | 1.21e-05 | 5.87e-07 | yes | yes | yes |
| verified|a3b_2ep vs a3b_base | 65 | 48 | +3.40 | 0.132 | 1.000 | 0.170 | no | no | no |
| verified|a3b_q36 vs q38 | 20 | 76 | -11.20 | 7.32e-09 | 6.37e-07 | 3.25e-08 | yes | yes | yes |
| verified|a3b_q36 vs n27 | 37 | 64 | -5.40 | 0.009 | 0.467 | 0.017 | yes | no | yes |
| verified|a3b_q36 vs s128 | 39 | 63 | -4.80 | 0.022 | 0.959 | 0.035 | yes | no | yes |
| verified|a3b_q36 vs s256 | 34 | 62 | -5.60 | 0.006 | 0.295 | 0.010 | yes | no | yes |
| verified|a3b_q36 vs s381 | 31 | 72 | -8.20 | 6.6e-05 | 0.005 | 0.000183 | yes | yes | yes |
| verified|a3b_q36 vs v13 | 38 | 54 | -3.20 | 0.117 | 1.000 | 0.155 | no | no | no |
| verified|a3b_q36 vs a3b_base | 90 | 35 | +11.00 | 9.24e-07 | 7.48e-05 | 3.31e-06 | yes | yes | yes |
| verified|a3b_q36 vs a3b_2ep | 75 | 37 | +7.60 | 0.000421 | 0.027 | 0.000994 | yes | yes | yes |
| verified|a3b_v12 vs q38 | 16 | 72 | -11.20 | 1.19e-09 | 1.08e-07 | 6.3e-09 | yes | yes | yes |
| verified|a3b_v12 vs n27 | 38 | 65 | -5.40 | 0.010 | 0.493 | 0.018 | yes | no | yes |
| verified|a3b_v12 vs s128 | 25 | 49 | -4.80 | 0.007 | 0.368 | 0.013 | yes | no | yes |
| verified|a3b_v12 vs s256 | 27 | 55 | -5.60 | 0.003 | 0.145 | 0.005 | yes | no | yes |
| verified|a3b_v12 vs s381 | 23 | 64 | -8.20 | 1.27e-05 | 0.000974 | 4.01e-05 | yes | yes | yes |
| verified|a3b_v12 vs v13 | 35 | 51 | -3.20 | 0.105 | 1.000 | 0.146 | no | no | no |
| verified|a3b_v12 vs a3b_base | 91 | 36 | +11.00 | 1.14e-06 | 9.12e-05 | 3.96e-06 | yes | yes | yes |
| verified|a3b_v12 vs a3b_2ep | 67 | 29 | +7.60 | 0.000132 | 0.009 | 0.000334 | yes | yes | yes |
| verified|a3b_v12 vs a3b_q36 | 45 | 45 | +0.00 | 1.000 | 1.000 | 1.000 | no | no | no |

## c. Minimum detectable effect

### c1. Paired design, from the observed discordance (k = 1 rollout): SE = sqrt((b+c) - (b-c)^2/N)/N, MDE_80 = 2.8 SE

**pro (N = 731)** — median SE 1.44 pp, median MDE_80 **4.03 pp** (median discordant pairs b+c = 111); over all 66 pro pairs the median MDE_80 is 4.07 pp.

| test (A - B) | b | c | b+c | SE pp | CI half-width pp | MDE_80 pp | N for MDE 3 pp (k = 1) |
|---|---|---|---|---|---|---|---|
| pro|s128 - n27 | 47 | 50 | 97 | 1.35 | 2.64 | 3.77 | 1156 |
| pro|s256 - n27 | 59 | 39 | 98 | 1.35 | 2.65 | 3.78 | 1162 |
| pro|s381 - n27 | 59 | 52 | 111 | 1.44 | 2.82 | 4.03 | 1322 |
| pro|v13 - n27 | 61 | 45 | 106 | 1.41 | 2.76 | 3.94 | 1259 |
| pro|a3b_2ep - a3b_base | 115 | 50 | 165 | 1.73 | 3.38 | 4.83 | 1898 |
| pro|a3b_v12 - a3b_q36 | 67 | 47 | 114 | 1.46 | 2.86 | 4.08 | 1352 |
| pro|a3b_v12t06 - a3b_q36t06 | 70 | 52 | 122 | 1.51 | 2.96 | 4.22 | 1449 |

N needed for MDE_80 = 3 pp at k = 1 from the observed discordance ((2.8/0.03)^2 x ((b+c)/N - delta^2)): median 1322 tasks.

**verified (N = 500)** — median SE 1.80 pp, median MDE_80 **5.04 pp** (median discordant pairs b+c = 81.0); over all 45 verified pairs the median MDE_80 is 5.36 pp.

| test (A - B) | b | c | b+c | SE pp | CI half-width pp | MDE_80 pp | N for MDE 3 pp (k = 1) |
|---|---|---|---|---|---|---|---|
| verified|s128 - n27 | 38 | 41 | 79 | 1.78 | 3.48 | 4.98 | 1377 |
| verified|s256 - n27 | 42 | 41 | 83 | 1.82 | 3.57 | 5.10 | 1447 |
| verified|s381 - n27 | 45 | 31 | 76 | 1.74 | 3.41 | 4.87 | 1318 |
| verified|v13 - n27 | 34 | 45 | 79 | 1.77 | 3.48 | 4.97 | 1373 |
| verified|a3b_2ep - a3b_base | 65 | 48 | 113 | 2.12 | 4.16 | 5.94 | 1959 |
| verified|a3b_v12 - a3b_q36 | 45 | 45 | 90 | 1.90 | 3.72 | 5.31 | 1568 |

N needed for MDE_80 = 3 pp at k = 1 from the observed discordance ((2.8/0.03)^2 x ((b+c)/N - delta^2)): median 1412 tasks.

### c2. Variance-decomposition MDE (rollout x k)

Inputs: sigma2_between_d = 0.0647, sigma2_within = 0.1322 (arm A) and 0.1378 (arm B); all three from the 237-task stratified rollout x3 subsample, not from the 731-task pool. Model: `Var(delta_hat) = [sigma2_between_d + (sigma2_within_A + sigma2_within_B)/k] / N ; MDE_80 = 2.8*sqrt(Var)` with sigma2_within_A + sigma2_within_B = 0.2700 (this is the reading that reproduces the narrative's "~4.1 pp at k = 3, N = 731"; treating the two values as alternative estimates of one pooled within-variance would give the smaller numbers listed below as a sensitivity line).

| N | k | per-task var of mean diff | SE pp | MDE_80 pp |
|---|---|---|---|---|
| 237 | 1 | 0.3347 | 3.76 | 10.52 |
| 237 | 3 | 0.1547 | 2.55 | 7.15 |
| 237 | 8 | 0.0984 | 2.04 | 5.71 |
| 237 | inf | 0.0647 | 1.65 | 4.63 |
| 500 | 1 | 0.3347 | 2.59 | 7.24 |
| 500 | 3 | 0.1547 | 1.76 | 4.93 |
| 500 | 8 | 0.0984 | 1.40 | 3.93 |
| 500 | inf | 0.0647 | 1.14 | 3.19 |
| 731 | 1 | 0.3347 | 2.14 | 5.99 |
| 731 | 3 | 0.1547 | 1.45 | 4.07 |
| 731 | 8 | 0.0984 | 1.16 | 3.25 |
| 731 | inf | 0.0647 | 0.94 | 2.63 |

- N needed for MDE_80 = 3 pp: k = 1: N = 2916, k = 3: N = 1348, k = 8: N = 858, k = inf: N = 564.
- **The earlier "4.6 pp floor" is 4.63 pp = 2.8 sqrt(0.0647/237): the k -> infinity value at N = 237, i.e. the size of the rollout x3 subsample, not of the evaluation pool.** At N = 731 the same between-task variance gives 2.63 pp (k -> inf) and 4.07 pp (k = 3).
- Sensitivity (alternative reading, not used): within = 0.1322, N = 731, k = 1: 4.60 pp; within = 0.1322, N = 731, k = 3: 3.42 pp; within = 0.1378, N = 731, k = 1: 4.66 pp; within = 0.1378, N = 731, k = 3: 3.44 pp.
- Consistency caveat: at k = 1 the decomposition implies a single-rollout discordance rate of about 0.335 (sigma2_between_d + within_A + within_B), whereas the observed (b+c)/N in the 731/500-task pools is 0.133-0.226 (baseline-relative pairs) and 0.114-0.133 for the same-weights temperature twins. The subsample therefore over-states the pool's run-to-run noise by roughly 2x in variance; the k = 1 row of c2 is an upper bound and the c1 numbers are the ones to quote for the actual design.

## d. Temperature twins on Pro (same weights, t = 0.8 vs t = 0.6; delta = t0.6 - t0.8)

| pair | b (t0.6 only) | c (t0.8 only) | flips b+c | flip rate (pool N) | flip rate (both present) | net delta pp | 95% CI pp | p exact |
|---|---|---|---|---|---|---|---|---|
| a3b_q36t06 vs a3b_q36 | 52 | 45 | 97 | 97/731 = 13.3% | 97/730 = 13.3% | +0.96 | [-1.68, +3.60] | 0.543 |
| a3b_v12t06 vs a3b_v12 | 44 | 39 | 83 | 83/731 = 11.4% | 82/722 = 11.4% | +0.68 | [-1.76, +3.13] | 0.661 |

### a3b_q36t06 vs a3b_q36: flips by language

| lang | n | flips | flip rate % | t0.6 only | t0.8 only | net delta pp | solved t0.8 | solved t0.6 | present in both |
|---|---|---|---|---|---|---|---|---|---|
| Go | 280 | 29 | 10.4 | 17 | 12 | +1.79 | 108 | 113 | 280 |
| JavaScript | 44 | 11 | 25.0 | 7 | 4 | +6.82 | 22 | 25 | 44 |
| Python | 266 | 37 | 13.9 | 18 | 19 | -0.38 | 145 | 144 | 265 |
| TypeScript | 141 | 20 | 14.2 | 10 | 10 | +0.00 | 62 | 62 | 141 |

### a3b_q36t06 vs a3b_q36: flips by repo

| repo | n | flips | flip rate % | t0.6 only | t0.8 only | net delta pp | solved t0.8 | solved t0.6 | present in both |
|---|---|---|---|---|---|---|---|---|---|
| ansible | 96 | 17 | 17.7 | 11 | 6 | +5.21 | 43 | 48 | 95 |
| openlibrary | 91 | 9 | 9.9 | 4 | 5 | -1.10 | 52 | 51 | 91 |
| flipt | 85 | 4 | 4.7 | 2 | 2 | +0.00 | 32 | 32 | 85 |
| qutebrowser | 79 | 11 | 13.9 | 3 | 8 | -6.33 | 50 | 45 | 79 |
| teleport | 76 | 9 | 11.8 | 4 | 5 | -1.32 | 29 | 28 | 76 |
| webclients | 65 | 6 | 9.2 | 3 | 3 | +0.00 | 33 | 33 | 65 |
| vuls | 62 | 7 | 11.3 | 5 | 2 | +4.84 | 28 | 31 | 62 |
| navidrome | 57 | 9 | 15.8 | 6 | 3 | +5.26 | 19 | 22 | 57 |
| element-web | 56 | 9 | 16.1 | 4 | 5 | -1.79 | 21 | 20 | 56 |
| NodeBB | 44 | 11 | 25.0 | 7 | 4 | +6.82 | 22 | 25 | 44 |
| tutanota | 20 | 5 | 25.0 | 3 | 2 | +5.00 | 8 | 9 | 20 |

### a3b_v12t06 vs a3b_v12: flips by language

| lang | n | flips | flip rate % | t0.6 only | t0.8 only | net delta pp | solved t0.8 | solved t0.6 | present in both |
|---|---|---|---|---|---|---|---|---|---|
| Go | 280 | 24 | 8.6 | 14 | 10 | +1.43 | 112 | 116 | 279 |
| JavaScript | 44 | 9 | 20.5 | 4 | 5 | -2.27 | 25 | 24 | 44 |
| Python | 266 | 33 | 12.4 | 15 | 18 | -1.13 | 147 | 144 | 258 |
| TypeScript | 141 | 17 | 12.1 | 11 | 6 | +3.55 | 73 | 78 | 141 |

### a3b_v12t06 vs a3b_v12: flips by repo

| repo | n | flips | flip rate % | t0.6 only | t0.8 only | net delta pp | solved t0.8 | solved t0.6 | present in both |
|---|---|---|---|---|---|---|---|---|---|
| ansible | 96 | 12 | 12.5 | 4 | 8 | -4.17 | 48 | 44 | 89 |
| openlibrary | 91 | 16 | 17.6 | 7 | 9 | -2.20 | 50 | 48 | 91 |
| flipt | 85 | 1 | 1.2 | 1 | 0 | +1.18 | 29 | 30 | 85 |
| qutebrowser | 79 | 5 | 6.3 | 4 | 1 | +3.80 | 49 | 52 | 78 |
| teleport | 76 | 9 | 11.8 | 7 | 2 | +6.58 | 29 | 34 | 75 |
| webclients | 65 | 6 | 9.2 | 4 | 2 | +3.08 | 35 | 37 | 65 |
| vuls | 62 | 7 | 11.3 | 2 | 5 | -4.84 | 33 | 30 | 62 |
| navidrome | 57 | 7 | 12.3 | 4 | 3 | +1.75 | 21 | 22 | 57 |
| element-web | 56 | 9 | 16.1 | 5 | 4 | +1.79 | 27 | 28 | 56 |
| NodeBB | 44 | 9 | 20.5 | 4 | 5 | -2.27 | 25 | 24 | 44 |
| tutanota | 20 | 2 | 10.0 | 2 | 0 | +10.00 | 11 | 13 | 20 |

## e. Decontamination slice: Pro without the teleport repo

`repo` values in the Pro rows: ansible (96), openlibrary (91), flipt (85), qutebrowser (79), teleport (76), webclients (65), vuls (62), navidrome (57), element-web (56), NodeBB (44), tutanota (20). The teleport repo is `teleport`: 76 tasks; Pro minus teleport = 655 tasks. (Verified contains no teleport tasks, so its tests are unchanged.)

### e1. Per-arm solved counts and rates (denominator = subset size; the "rows" columns use only the tasks the arm actually ran, as the narrative's 37.3 -> 47.4 % figures do)

| arm | teleport solved/76 | rate % | rows | rate % (rows) | rest solved/655 | rate % | rows | rate % (rows) | all solved/731 | rate % |
|---|---|---|---|---|---|---|---|---|---|---|
| q38 | 36 | 47.4 | 76 | 47.4 | 371 | 56.6 | 654 | 56.7 | 407 | 55.7 |
| n27 | 28 | 36.8 | 75 | 37.3 | 338 | 51.6 | 650 | 52.0 | 366 | 50.1 |
| s128 | 33 | 43.4 | 76 | 43.4 | 330 | 50.4 | 654 | 50.5 | 363 | 49.7 |
| s256 | 31 | 40.8 | 75 | 41.3 | 355 | 54.2 | 647 | 54.9 | 386 | 52.8 |
| s381 | 36 | 47.4 | 76 | 47.4 | 337 | 51.5 | 651 | 51.8 | 373 | 51.0 |
| v13 | 31 | 40.8 | 76 | 40.8 | 351 | 53.6 | 655 | 53.6 | 382 | 52.3 |
| a3b_base | 22 | 28.9 | 76 | 28.9 | 237 | 36.2 | 654 | 36.2 | 259 | 35.4 |
| a3b_2ep | 24 | 31.6 | 72 | 33.3 | 300 | 45.8 | 650 | 46.2 | 324 | 44.3 |
| a3b_q36 | 29 | 38.2 | 76 | 38.2 | 308 | 47.0 | 655 | 47.0 | 337 | 46.1 |
| a3b_q36t06 | 28 | 36.8 | 76 | 36.8 | 316 | 48.2 | 654 | 48.3 | 344 | 47.1 |
| a3b_v12 | 29 | 38.2 | 76 | 38.2 | 328 | 50.1 | 655 | 50.1 | 357 | 48.8 |
| a3b_v12t06 | 34 | 44.7 | 75 | 45.3 | 328 | 50.1 | 647 | 50.7 | 362 | 49.5 |

### e2. The 7 Pro baseline-relative tests on Pro minus teleport (N = 655), Holm within these 7 (m = 7)

delta = (b - c)/655.

| test (A - B) | b (A only) | c (B only) | delta pp | 95% CI pp | p exact | p Holm | q BH | raw<.05 | Holm<.05 | BH<.05 |
|---|---|---|---|---|---|---|---|---|---|---|
| pro-minus-teleport|s128 - n27 | 40 | 48 | -1.22 | [-4.03, +1.58] | 0.456 | 0.911 | 0.532 | no | no | no |
| pro-minus-teleport|s256 - n27 | 52 | 35 | +2.60 | [-0.19, +5.38] | 0.086 | 0.428 | 0.200 | no | no | no |
| pro-minus-teleport|s381 - n27 | 51 | 52 | -0.15 | [-3.19, +2.88] | 1.000 | 1.000 | 1.000 | no | no | no |
| pro-minus-teleport|v13 - n27 | 54 | 41 | +1.98 | [-0.93, +4.90] | 0.218 | 0.872 | 0.382 | no | no | no |
| pro-minus-teleport|a3b_2ep - a3b_base | 104 | 41 | +9.62 | [+6.09, +13.15] | 1.7e-07 | 1.19e-06 | 1.19e-06 | yes | yes | yes |
| pro-minus-teleport|a3b_v12 - a3b_q36 | 59 | 39 | +3.05 | [+0.10, +6.01] | 0.054 | 0.326 | 0.190 | no | no | no |
| pro-minus-teleport|a3b_v12t06 - a3b_q36t06 | 61 | 49 | +1.83 | [-1.30, +4.97] | 0.294 | 0.883 | 0.412 | no | no | no |

- survive raw 0.05: pro-minus-teleport|a3b_2ep - a3b_base
- survive Holm 0.05: pro-minus-teleport|a3b_2ep - a3b_base
- survive BH 0.05: pro-minus-teleport|a3b_2ep - a3b_base

### e3. The 13-test family with the 7 Pro tests replaced by their teleport-excluded versions (Verified unchanged), Holm within 13 (m = 13)

| test (A - B) | b (A only) | c (B only) | delta pp | 95% CI pp | p exact | p Holm | q BH | raw<.05 | Holm<.05 | BH<.05 |
|---|---|---|---|---|---|---|---|---|---|---|
| pro-minus-teleport|s128 - n27 | 40 | 48 | -1.22 | [-4.03, +1.58] | 0.456 | 1.000 | 0.658 | no | no | no |
| pro-minus-teleport|s256 - n27 | 52 | 35 | +2.60 | [-0.19, +5.38] | 0.086 | 0.943 | 0.352 | no | no | no |
| pro-minus-teleport|s381 - n27 | 51 | 52 | -0.15 | [-3.19, +2.88] | 1.000 | 1.000 | 1.000 | no | no | no |
| pro-minus-teleport|v13 - n27 | 54 | 41 | +1.98 | [-0.93, +4.90] | 0.218 | 1.000 | 0.472 | no | no | no |
| pro-minus-teleport|a3b_2ep - a3b_base | 104 | 41 | +9.62 | [+6.09, +13.15] | 1.7e-07 | 2.21e-06 | 2.21e-06 | yes | yes | yes |
| pro-minus-teleport|a3b_v12 - a3b_q36 | 59 | 39 | +3.05 | [+0.10, +6.01] | 0.054 | 0.653 | 0.352 | no | no | no |
| pro-minus-teleport|a3b_v12t06 - a3b_q36t06 | 61 | 49 | +1.83 | [-1.30, +4.97] | 0.294 | 1.000 | 0.478 | no | no | no |
| verified|s128 - n27 | 38 | 41 | -0.60 | [-4.08, +2.88] | 0.822 | 1.000 | 1.000 | no | no | no |
| verified|s256 - n27 | 42 | 41 | +0.20 | [-3.37, +3.77] | 1.000 | 1.000 | 1.000 | no | no | no |
| verified|s381 - n27 | 45 | 31 | +2.80 | [-0.61, +6.21] | 0.135 | 1.000 | 0.352 | no | no | no |
| verified|v13 - n27 | 34 | 45 | -2.20 | [-5.68, +1.28] | 0.260 | 1.000 | 0.478 | no | no | no |
| verified|a3b_2ep - a3b_base | 65 | 48 | +3.40 | [-0.76, +7.56] | 0.132 | 1.000 | 0.352 | no | no | no |
| verified|a3b_v12 - a3b_q36 | 45 | 45 | +0.00 | [-3.72, +3.72] | 1.000 | 1.000 | 1.000 | no | no | no |

- survive raw 0.05: pro-minus-teleport|a3b_2ep - a3b_base
- survive Holm 0.05: pro-minus-teleport|a3b_2ep - a3b_base
- survive BH 0.05: pro-minus-teleport|a3b_2ep - a3b_base

### e4. Same 7 tests on teleport only (N = 76), for reference (m = 7)

delta = (b - c)/76.

| test (A - B) | b (A only) | c (B only) | delta pp | 95% CI pp | p exact | p Holm | q BH | raw<.05 | Holm<.05 | BH<.05 |
|---|---|---|---|---|---|---|---|---|---|---|
| pro-teleport|s128 - n27 | 7 | 2 | +6.58 | [-1.02, +14.17] | 0.180 | 0.898 | 0.419 | no | no | no |
| pro-teleport|s256 - n27 | 7 | 4 | +3.95 | [-4.56, +12.45] | 0.549 | 1.000 | 0.768 | no | no | no |
| pro-teleport|s381 - n27 | 8 | 0 | +10.53 | [+3.63, +17.43] | 0.008 | 0.055 | 0.055 | yes | no | no |
| pro-teleport|v13 - n27 | 7 | 4 | +3.95 | [-4.56, +12.45] | 0.549 | 1.000 | 0.768 | no | no | no |
| pro-teleport|a3b_2ep - a3b_base | 11 | 9 | +2.63 | [-8.89, +14.15] | 0.824 | 1.000 | 0.961 | no | no | no |
| pro-teleport|a3b_v12 - a3b_q36 | 8 | 8 | +0.00 | [-10.32, +10.32] | 1.000 | 1.000 | 1.000 | no | no | no |
| pro-teleport|a3b_v12t06 - a3b_q36t06 | 9 | 3 | +7.89 | [-0.86, +16.65] | 0.146 | 0.876 | 0.419 | no | no | no |

- survive raw 0.05: pro-teleport|s381 - n27
- survive Holm 0.05: none
- survive BH 0.05: none

### e5. Per-repo table, Pro (n tasks; solved per arm, missing = unsolved)

| repo | lang | n | q38 | n27 | s128 | s256 | s381 | v13 | a3b_base | a3b_2ep | a3b_q36 | a3b_q36t06 | a3b_v12 | a3b_v12t06 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ansible | Python | 96 | 51 | 51 | 49 | 53 | 49 | 49 | 40 | 48 | 43 | 48 | 48 | 44 |
| openlibrary | Python | 91 | 57 | 51 | 49 | 54 | 49 | 51 | 36 | 43 | 52 | 51 | 50 | 48 |
| flipt | Go | 85 | 31 | 30 | 28 | 29 | 31 | 31 | 24 | 27 | 32 | 32 | 29 | 30 |
| qutebrowser | Python | 79 | 56 | 50 | 49 | 50 | 49 | 52 | 34 | 40 | 50 | 45 | 49 | 52 |
| teleport | Go | 76 | 36 | 28 | 33 | 31 | 36 | 31 | 22 | 24 | 29 | 28 | 29 | 34 |
| webclients | TypeScript | 65 | 38 | 37 | 34 | 38 | 33 | 38 | 28 | 35 | 33 | 33 | 35 | 37 |
| vuls | Go | 62 | 38 | 34 | 31 | 34 | 29 | 35 | 25 | 30 | 28 | 31 | 33 | 30 |
| navidrome | Go | 57 | 26 | 23 | 22 | 27 | 26 | 27 | 16 | 23 | 19 | 22 | 21 | 22 |
| element-web | TypeScript | 56 | 28 | 23 | 28 | 29 | 29 | 31 | 15 | 22 | 21 | 20 | 27 | 28 |
| NodeBB | JavaScript | 44 | 32 | 27 | 26 | 27 | 29 | 24 | 12 | 22 | 22 | 25 | 25 | 24 |
| tutanota | TypeScript | 20 | 14 | 12 | 14 | 14 | 13 | 13 | 7 | 10 | 8 | 9 | 11 | 13 |

## f. What changes for the paper (NARRATIVE_REPORT.md section H2 / H4)

- **Nothing in report.json is wrong**: all 111 valid pairs reproduce b, c, delta, the Wald CI (z = 1.96) and the exact McNemar p to 1e-9, so every raw number quoted in H2/H4 stands; what changes is the wording around corrections, the MDE, and the decontamination slice.
- **H2, pre-declared corpus family (13 tests)**: exactly one test survives Holm and BH at 0.05 — pro|a3b_2ep - a3b_base (+8.89 pp, p = 4.51e-07, Holm p = 5.87e-06). Every post-trained-base comparison is null after correction: s256 - n27 Pro +2.74 pp has exact p = 0.054, Holm-13 p = 0.653, BH q = 0.269; a3b_v12 - a3b_q36 Pro +2.74 pp has p = 0.075, Holm-13 p = 0.822. The sentence 's256 vs n27 on Pro is p = 0.054 with a Wald CI that just excludes zero' must not be read as evidence: the CI [+0.09, +5.38] excludes zero only through the normal approximation while the exact test does not reject; report the exact p as the decision and drop or re-label the Wald CI where the two disagree (same situation for a3b_v12 - a3b_q36 on Pro-minus-teleport: CI [+0.10, +6.01], p = 0.054).
- **H2, the two 'exceptions'** (v13 vs s381 Verified -5.00 pp, p = 0.005; s256 vs s128 Pro +3.15 pp, p = 0.014) do not survive Holm over the 111 valid tests (Holm p = 0.250 and 0.658); they do survive BH (q = 0.009 and 0.024). Neither is a pre-declared endpoint (both compare checkpoints with each other), so word them as exploratory, 'significant before correction, retained at FDR 0.05, not at FWER 0.05'. The v13-mask cost of -5.0 pp on Verified (H4, reconciliations) carries the same qualifier.
- **H2, generation jump**: q38 - n27 survives Holm within the 6-test generation/architecture family on both pools (Pro +5.61 pp, Holm p = 0.000537; Verified +5.80 pp, Holm p = 0.009). Over the 111-test family only the Pro test survives Holm (adj. p = 0.007); Verified has Holm-111 p = 0.127 but BH-111 q = 0.005. Say which family the correction is over; recommended wording: 'significant after Holm within the pre-declared family on both pools; after Holm over all 111 tests on Pro only'.
- **H2, dense vs sparse**: n27 - a3b_q36 survives Holm within the 6-test family on both pools (Pro +3.97 pp, Holm p = 0.023; Verified +5.40 pp, Holm p = 0.023) but not Holm-111 (adj. p = 0.387 / 0.467; BH-111 q = 0.014 / 0.017). After the same corpus, s381 - a3b_v12 is +2.19 pp (p = 0.137) on Pro — null under every correction — and +8.20 pp on Verified, which survives everything (Holm-111 p = 0.000974). 'Score follows the architecture' should therefore be stated as: same sign on both pools, established on Verified, not resolved on Pro.
- **H2, 'Holm/FDR over the 111 paired tests'** (the announced fix): 74 of 111 tests are raw-significant, 49 survive Holm, 71 survive BH; the 25 tests that Holm removes are all |delta| <= 5.8 pp (raw p >= 0.0011). Suggested policy for the paper: pre-declared families with Holm as the primary decision rule (13 corpus tests; 6 generation/architecture tests), Holm over all 111 as a sensitivity, BH only as a screening statement; one p-value per test.
- **H2, decontamination / teleport** (repo value `teleport`, 76/731 tasks; Pro-minus-teleport = 655): the narrative's 's381 gain sits entirely in teleport (37.3 -> 47.4 % vs 52.0 -> 51.8 % elsewhere)' is confirmed with per-arm-rows denominators; with the pool convention it reads 36.8 -> 47.4 % on teleport vs 51.6 -> 51.5 % elsewhere. On Pro-minus-teleport (N = 655): s128 -1.22 (p = 0.456), s256 +2.60 [-0.19, +5.38] (p = 0.086), s381 -0.15 (p = 1.000), v13 +1.98 (p = 0.218) vs n27; a3b_v12 - a3b_q36 +3.05 (p = 0.054); a3b_2ep - a3b_base +9.62 [+6.09, +13.15] (p = 1.7e-07); temperature-0.6 pair +1.83 (p = 0.294). Holm within these 7 (and within the 13 with Verified unchanged) again leaves only a3b_2ep - a3b_base. Decontamination statement to write: excluding teleport changes no decision (nothing becomes significant, nothing significant becomes null); it removes s381's apparent Pro gain (+0.96 -> -0.15 pp) and slightly strengthens s256 (+2.74 -> +2.60) and a3b_v12 (+2.74 -> +3.05), i.e. the post-trained-base corpus effect is not a teleport artefact — it is simply below the MDE everywhere.
- **H2, teleport as a repo-level (not instance-level) advantage**: on the 76 teleport tasks every v12/v13 checkpoint beats n27 — s128 +6.58, s256 +3.95, s381 +10.53 (b = 8, c = 0, p = 0.008, Holm-7 p = 0.055), v13 +3.95 pp — while elsewhere the same checkpoints are -1.2 to +2.6 pp. This is consistent with the corpus's 1,622 teleport trajectories helping on the repo even though instance-level matches are zero; the paper should say so and keep the Pro-minus-teleport table as the primary Pro table for corpus effects. Caveat for the same paragraph: a 76-task slice has SE about 4 pp and the same-weights twins flip 9/9 teleport tasks between temperatures, so 'entirely in teleport' is a one-rollout observation, not a measured effect.
- **H4, run-to-run**: the quoted flips are exact — 97/731 = 13.3 % (a3b_q36) and 83/731 = 11.4 % (a3b_v12), net +0.96 pp (p = 0.543) and +0.68 pp (p = 0.661); restricting to tasks present in both runs gives the same rates (97/730, 82/722), so the missing-task convention is not driving them. New material to add: flip rate by language ranges from Go 10.4 % (n = 280) / Go 8.6 % (n = 280) to JavaScript 25.0 % (n = 44) / JavaScript 20.5 % (n = 44); by repo from flipt 4.7 % (n = 85) / flipt 1.2 % (n = 85) to NodeBB 25.0 % (n = 44) / NodeBB 20.5 % (n = 44) — the noise is not uniform across strata, which matters for any per-language claim.
- **H4, MDE — replace the 3.8 pp / 4.6 pp sentences**: (1) the '4.6 pp floor' is 4.63 pp = 2.8 sqrt(0.0647/237), the k -> infinity value at N = 237 (the rollout x3 subsample size); at N = 731 the same between-task variance gives 2.63 pp (k -> inf), 3.25 pp (k = 8), 4.07 pp (k = 3) and 5.99 pp (k = 1); at N = 500: 3.19 / 4.93 / 7.24 pp (k = inf / 3 / 1). (2) The '3.8 pp' figure is one pair (s256 - n27 on Pro); the median one-rollout paired MDE_80 over the pre-declared baseline-relative pairs is 4.03 pp on Pro (range 3.77-4.83) and 5.04 pp on Verified (range 4.87-5.94); quote these. (3) N for MDE_80 = 3 pp: from the observed one-rollout discordance about 1322 Pro tasks (k = 1); from the decomposition 1348 tasks at k = 3 and 564 at k -> inf. (4) State the provenance: the decomposition numbers are from the 237-task stratified rollout x3 subsample, and its implied single-rollout discordance (0.33) is about twice what the 731/500-task pools show (0.13-0.23; twins 0.11-0.13), so the k = 1 decomposition row is an upper bound; use the observed-discordance MDE (c1) for the actual design and the decomposition only for the k > 1 projections, labelled conservative.
- **H4 accepted wording** ('one-rollout paired pass@1 does not resolve most observed 0-3 pp SFT differences') is consistent with the corrected MDE (about 4 pp Pro, 5 pp Verified) and can stay; the 'Fixes' bullet of H2 (Holm/FDR, pre-declared endpoints, decontamination table) is now delivered by stats_fixes.md sections b, c and e and should be cited from there.