A_evaluation_noise · 完整稿件源码

% A_evaluation_noise.tex -- Appendix A (HIGH priority). All p-values, Holm families, MDE, flips, teleport
% slices and rollout x3 live here, never in the body (presentation rule).
% Numbers: analysis/stats/stats_fixes.md / stats_fixes.json (Tables A1-A6 generated by tables/make_tabA.py);
% NARRATIVE_REPORT.md H4 for the rollout x3 lines; analysis/annot/second_annotator_agreement.md (Table A7);
% Table 3 (tab:endings) generated by tables/make_tab3.py from bench/scripts/paper/out/pp_data.json.
\section{Evaluation noise}
\label{app:noise}

\paragraph{Conventions.}
Every score-level statistic in this appendix is recomputed with the standard library from \code{data/report.json} (\code{analysis/stats/stats\_fixes.py}).
The pool is the union of task ids over the arms of that pool (731 on \SWEPro, 500 on \SWEVerified) and is the denominator of every rate and difference; a task missing from an arm counts as unsolved (0--14 missing per arm and pool, Table~\ref{tab:C2}; scoring each arm on the tasks it ran raises \passat by 0.0--2.1\pp, most for \sThreeEightOne on Verified, and leaves every comparison on the same side of its detectable effect).
For a pair $A$ vs $B$, $b$ is the number of tasks solved by $A$ only, $c$ by $B$ only, $\Delta=(b-c)/N$, and $p$ is the two-sided exact binomial McNemar test on the $b+c$ discordant tasks.
Of the 132 pairs in \code{report.json}, 21 involve the repeat-sample arms on Verified, which were never run, and are void; the remaining 111 reproduce \code{report.json} exactly in $b$, $c$, $\Delta$ and $p$.

\paragraph{Pre-declared paired tests.}
Table~\ref{tab:A1} lists the two pre-declared families with Holm correction as the primary decision rule: the 13 baseline-relative corpus tests (each fine-tuned arm against its own starting model) and the 6 generation/architecture tests.
Exactly one corpus test survives Holm, \aTwoEp $-$ \aBase on Pro ($+8.89$\pp, Holm $p=5.87\times10^{-6}$); every post-trained-base comparison is null after correction, including \sTwoFiveSix $-$ \nTwoSeven on Pro (exact $p=0.054$, Holm $p=0.653$), whose Wald interval $[+0.09,\,+5.38]$ excludes zero only through the normal approximation.
Five of the six generation/architecture tests survive Holm; \sThreeEightOne $-$ \aV on Pro ($+2.19$\pp, $p=0.137$) does not.
Over all 111 valid pairs, 74 are significant before correction, 49 after Holm and 71 after Benjamini--Hochberg; the two nominal checkpoint-versus-checkpoint exceptions, \vThirteen vs \sThreeEightOne on Verified ($-5.00$\pp, $p=0.005$) and \sTwoFiveSix vs \sOneTwoEight on Pro ($+3.15$\pp, $p=0.014$), survive Benjamini--Hochberg only (Holm-111 $p=0.250$ and $0.658$) and are not pre-declared endpoints.

\input{tables/tabA1_paired}

\paragraph{Detectable effect.}
Table~\ref{tab:A2} derives the minimum detectable effect of the one-rollout paired design \citep{miller2024adding,zhuang2026preregistering} from the observed discordance of the baseline-relative pairs: the median is 4.03\pp on Pro (range 3.77--4.83) and 5.04\pp on Verified (4.87--5.94), and about 1,322 Pro tasks would be needed to resolve 3\pp at one rollout.
Table~\ref{tab:A3} projects the effect of more rollouts per task from a variance decomposition estimated on the 237-task stratified rollout$\times$3 subsample: at $N=731$ it gives 5.99 / 4.07 / 3.25 / 2.63\pp for $k=1$ / 3 / 8 / $\infty$.
Because the subsample over-states the pools' single-rollout discordance by about a factor of two in variance, the $k=1$ projection is an upper bound and Table~\ref{tab:A2} is the figure to quote for the actual design.

\input{tables/tabA2_mde}
\input{tables/tabA3_mde_k}

\begin{figure*}[t]
  \centering
  \IfFileExists{../figures/fig13.pdf}{\includegraphics[width=\textwidth]{../figures/fig13.pdf}}{\fbox{\parbox[c][4cm][c]{0.95\textwidth}{\centering Placeholder (\code{figures/fig13.pdf}).}}}
  \caption{Robustness of the score tables on \SWEPro. \textbf{(a)}~Repeat samples: share of tasks solved in exactly one of two runs of the same weights, overall (bars), per language (symbols sized by $n$) and per repository (grey dots). \textbf{(b)}~Teleport slice: \passat on the 76 teleport tasks (open) against the other 655 (filled) for the twelve arms, labelled with the difference in pp. Neither a second run nor the teleport repository reorders the arms.}
  \label{fig:robust}
\end{figure*}
\paragraph{Run-to-run noise.}
Figure~\ref{fig:robust} draws the flip rates and the teleport slice; 
Table~\ref{tab:A4} gives the two same-weights pairs on Pro: a second run of \aQ flips 97 of 731 task verdicts (13.3\,\%) for a net $+0.96$\pp ($p=0.543$), and \aV flips 83 (11.4\,\%) for $+0.68$\pp ($p=0.661$).
The flip rate ranges from Go (10.4 / 8.6\,\%) to JavaScript (25.0 / 20.5\,\%) and, by repository, from \code{flipt} (4.7 / 1.2\,\%) to \code{NodeBB} (25.0 / 20.5\,\%), so the noise is not uniform across strata.

\input{tables/tabA4_twins}

\paragraph{Three rollouts.}
% Source: NARRATIVE_REPORT.md H4 (rollout x3 lines; computed by bench/scripts 算_rollout稳定性.py for the 27B report page).
% The 237-task subsample is the one whose variance components feed Table A3 (stats_fixes.md section c2).
On the 237-task stratified subsample replayed three times, 36--42\,\% of Pro tasks receive inconsistent verdicts across the three rollouts, and the stratified re-estimate of \sTwoFiveSix $-$ \nTwoSeven is $+1.40$ / $+2.75$\pp, not distinguishable from zero.

\paragraph{Teleport slice.}
The corpus contains 1,622 teacher trajectories from the \code{teleport} repository, which supplies 76 of the 731 Pro tasks (Verified has none).
Table~\ref{tab:A5} gives every arm's solved counts on the 76 teleport tasks and on the other 655, and Table~\ref{tab:A6} repeats the seven Pro baseline-relative tests on each slice.
Excluding teleport changes no decision: nothing becomes significant and nothing significant becomes null.
It removes \sThreeEightOne's apparent Pro gain ($+0.96\to-0.15$\pp) and slightly strengthens \sTwoFiveSix ($+2.74\to+2.60$\pp) and \aV ($+2.74\to+3.05$\pp), so the post-trained-base corpus effect is not a teleport artefact but is below the detectable effect everywhere.
On the 76 teleport tasks alone every \vTwelve/\vThirteenCorpus checkpoint beats \nTwoSeven, by $+3.9$ to $+10.5$\pp (\sThreeEightOne: 8 vs 0 discordant, $p=0.008$, Holm-7 $p=0.055$), a repository-level lift consistent with corpus exposure although instance-level matches are zero (Appendix~\ref{app:extras}); a 76-task slice has a standard error of about 4\pp and the repeat runs flip 9 of its tasks, so this is a one-rollout observation rather than a measured effect.

\input{tables/tabA5_teleport_arms}
\input{tables/tabA6_teleport_tests}

\paragraph{Endings audit.}
Figure~\ref{fig:endings} draws the regex rates per arm and 
Table~\ref{tab:endings} gives the regex rates behind \S\ref{sec:measurement}.
Two model annotators read the same 180 endings (110 regex-negative, 70 regex-positive) and the 68 rationalization hits, the second (a different model family) blind to the first's labels (Table~\ref{tab:A7}).
They agree on the two quantities the body uses: 0 of 110 regex-negative failure endings admit failure, and the strong-form regex precision is 69 of 70.
They differ on one \qThreeEight sample (claim versus truncated) and on the split of the rationalization hits (48 versus 49 verified attributions, 12 versus 6 bare dismissals, 8 versus 11 false positives); the verified-attribution share on which they agree (\nTwoSeven 2/8, \sTwoFiveSix 12--13/15, teacher 10--11/15) is the only rationalization statistic the paper reports.
No human pass was run.

\input{tables/tab3_endings}
\input{tables/tabA7_annot}

\clearpage