07_measurement · 完整稿件源码

% 07_measurement.tex -- section 7, budget 0.3 page + Fig. 6 (endings). H4 as a methodological finding about the
% evaluation (not a confession): run-to-run noise, detectable effect, and what the success-claim regex measures.
% Statistics in Appendix A. Reframed 2026-09-02 at the user's request; every number is unchanged.
\section{What the score tables can resolve}
\label{sec:measurement}
\begin{figure}[t]
  \centering
  \IfFileExists{../figures/fig7.pdf}{\includegraphics[width=0.66\columnwidth]{../figures/fig7.pdf}}{\fbox{\parbox[c][5cm][c]{0.95\columnwidth}{\centering Placeholder for Fig.~7 (\code{figures/fig7.pdf}).}}}
  \caption{Endings of unsolved trajectories: share whose final message matches the strong-form completion regex (solid) and the rationalization regex (light) per arm on \SWEPro (top) and \SWEVerified (bottom); dashed and dotted lines = the teacher's failed by-product trajectories (44.6 and 7.0\,\%, measured on the teacher's own task pool). \vTwelve moves the dense line's strong-form share toward the teacher on Pro; Verified is lower and not monotone; audited endings that escape the regex assert completion in variant wording in every arm.}
  \label{fig:endings}
\end{figure}

Three properties of the evaluation bound the score tables.
First, single-rollout outcomes are noisy: two runs of the same weights flip 13\,\% of \SWEPro tasks for \aQ and 11\,\% for \aV, for net differences below 1\pp, the run-to-run reference for every paired comparison \citep{wang2026contractbench,yagubyan2026how}.
Second, the paired design resolves differences of about 4\pp on Pro and 5\pp on Verified (\S\ref{sec:setup}), so the 0--3\pp differences between post-trained bases and their students are read as unresolved, whereas the behavioral rulers separate the same arms by seven to thirty times their sampling floors (Appendix~\ref{app:fp}): at this scale the fingerprint is the sensitive readout, as interface perturbation is for agent benchmarks \citep{gu2026what}.
Third, the success-claim regex tracks completion phrasing rather than outcome (\S\ref{sec:corpus}; \citealp{advani2026from}).
On \SWEPro the share of unsolved trajectories whose final message asserts completion in the strong form falls from 66\,\% (\nTwoSeven) to 49--53\,\% after \vTwelve and 44\,\% after \vThirteenCorpus, level with the teacher's 45\,\%, so distillation moves this phrasing toward the teacher; on Verified the shares are lower and not monotone (Figure~\ref{fig:endings}; Table~\ref{tab:endings}).
The 110 audited endings that escape the regex assert completion in variant wording in every arm, teacher included (Appendix~\ref{app:noise}).