\section{Evaluation}
\label{sec:evaluation}
\paragraph{Policy effectiveness: pending.}
\Cref{tab:results} reserves the prescribed models, training seeds, and task partitions. None of these policy comparisons has yet been measured. The primary M--B contrast tests allocation of the extra feedback on the designated holdout; M--O and the single-seed M42--E42 comparison address the corresponding terminal-feedback alternatives.
\input{../figures/TABLE_counterfactual_results_pending}
\paragraph{Does artifact removal produce a usable signal?}
The base-model development pilot generated eight trajectories on each of the original 40 tasks. Of these 320 planned trajectories, 244 met the training query rules and all 244 intervention trials returned usable reports. Here, usability follows the training adapter's fixed-list utility contract. The exclusions were 41 format-error trajectories, 34 without an eligible block, and one without an original utility after the agent deleted installation files. This last trajectory is retained as a model failure; it is not resampled.
\input{../figures/TABLE_counterfactual_pilot}
The contrast is nonzero for 92 queries (37.7\%), including 17 from failed original trajectories: 12 positive and five negative, spanning 14 tasks and six repositories. These are 17 of the 111 queried failed trajectories, within 150 original trajectories with a known failed verdict. The twelve positive contrasts split evenly between a positive fail-to-pass (F2P) component and the regression indicator alone (\cref{app:signal-origin}). These observations establish information availability on failures, rather than a benefit from learning with that information. \Cref{tab:pilot} preserves the original pilot denominator and separately exposes the 39-task stratum admitted to training.
The cap activates for 83 of the 92 nonzero contrasts (90.2\%), placing most observed nonzero cases in the sign-and-support regime of \cref{eq:coefficients}. Among these 92 cases, the median selected-token fraction $m/L$ is 2.10\% and median shrinkage $s$ is 0.181; \cref{app:geometry} gives their distributions. Before format-error and original-utility query exclusions, 282 trajectories have a constructed intervention artifact; of these, 47 become an empty patch. The median fraction of final changed paths touched by the selected block is 0.5. One of the 92 nonzero cases instead selects paths absent from the final patch: the block repairs earlier agent damage. This illustrates why last-touch eligibility is defined relative to the pre-block tree rather than only to the final net diff. These structural measurements do not certify semantic attribution accuracy.
\paragraph{Is verification repeatable, and what does it cost?}
All 120 prescribed cross-machine factual replays match their originals at the level of reported test labels. The initial 244 intervention trials take 237.3 seconds on average, with a 416.9-second empirical 95th percentile. A separate deployment repeat of the same 244 artifacts on the designated verifier host, at concurrency 24 with images retained between trials, yields identical per-test reports for all cases and takes 115.1 seconds on average, with a 189.9-second 95th percentile. Service time includes setup, verification, and cleanup, and excludes semaphore waiting. Host placement, concurrency, and cleanup differ together; the comparison does not isolate a cache effect. Production queueing and total training cost remain unmeasured.
\paragraph{Does the coefficient interface preserve its intended behavior?}
The admitted 149-task profile has 298 gold/unchanged-tree controls with the expected resolved outcomes. Twenty CPU checks exercise the auxiliary interface, including batch concatenation and padding, group advantages, prompt-mean reduction, and zero-credit DPPO loss/gradient equivalence. Separate nonzero coefficient and backward checks validate the M/B construction. \Cref{app:validation} specifies this evidence. Distributed nonzero-credit learning and the fixed-endpoint policy comparisons remain to be evaluated.
完整资料 / R1
现稿评测:pilot、成本与效果Pending
paper-rl/paper/sections/5_evaluation.tex
公开内容 SHA-256 90a6a8574b27466b6e876253200ff9e7762f5747fc8c62cfe4cb0e71484bc811