5_experiments · 完整稿件源码

\section{Preliminary Evidence and Evaluation}
\label{sec:experiments}
\paragraph{Completed CPU study.}
The three current-correctness LoRA runs completed three epochs each using eight CPU threads and no GPU allocation. They train on the same 128 states and evaluate on the same 32 development states from eight tasks. Each run took 1.91--2.05 process hours. Their final accuracies are 78.1\%, 71.9\%, and 81.3\%; mean accuracy is 77.1\% with a sample standard deviation of 4.8 percentage points. These checkpoints predict current correctness, not action utility.

\begin{table}[t]
\centering\small
\caption{Current-correctness development results. LoRA reports the mean across three seeds; other rows are single fitted or frozen baselines. All rows evaluate the same 32 states.}
\begin{tabular}{@{}lrrr@{}}
\toprule
Method & Acc.\,\(\uparrow\) & Brier\,\(\downarrow\) & AUC\,\(\uparrow\)\\
\midrule
Train prior & .531 & .249 & .500\\
Frozen 0.6B & .469 & .510 & .522\\
TF--IDF + GBDT & .531 & .342 & .494\\
TF--IDF logistic & .719 & .213 & .804\\
\textbf{CPU LoRA (3 seeds)} & \textbf{.771} & \textbf{.184} & \textbf{.846}\\
\bottomrule
\end{tabular}
\label{tab:current}
\end{table}


Figure~\ref{fig:cpu_baselines} shows descriptive task-cluster intervals and individual LoRA seed estimates. The mean Brier reduction relative to logistic regression is 0.0285, with a paired task-bootstrap 95\% interval of $[-0.0639,0.0077]$ for LoRA minus logistic. The interval crosses zero. These development results motivate reusing the correctness model, but do not establish a reliable advantage on new tasks or an improvement in agent control. Full CPU learning curves are given in Appendix~\ref{sec:cpu_measurements}.

\paragraph{Execution integrity.}
All 160 training and development states have public-test menus. The initial 12-state audit reproduces every original official patch verdict. Its 24 public-test calls complete: 22 pass and two return test failures. Two states pass both selected public checks while failing the official evaluator. Public feedback therefore remains separate from the correctness label. The audit's median public-test service time is 1.728 seconds.

\paragraph{Controller overhead.}
A CPU forward pass of the 0.6B architecture takes 0.728, 3.374, and 8.788 seconds at 512, 2,048, and 4,096 input tokens, respectively. These are three-repeat medians on eight CPU threads in FP32. Thus a long-input controller can cost more wall time than a selected public test. Any efficiency claim must include decision overhead and should be tested against avoided repair or retry work, not inferred from fewer test calls. The complete hardware protocol appears in Appendix~\ref{sec:cpu_measurements}.

\paragraph{Completed final-state action pilot.}
The frozen 12-state pilot contains 11 training tasks. All 108 executions receive official verdicts: one submission check per state and two seeds for each of four nonterminal actions. Non-restart initial trees match the corresponding submitted candidate, and no selected public test is modified. Table~\ref{tab:action_pilot} reports descriptive branch counts, not a learned-controller comparison.

\begin{table}[t]
\centering\small
\caption{G001 final-state pilot. Resolution is the fraction of officially successful executions. Repair counts condition on the three initially unsuccessful candidates (two seeds each). Token means count added outputs only.}
\input{../figures/TABLE_action_pilot}
\label{tab:action_pilot}
\end{table}

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{../figures/fig5_action_pilot.pdf}
\caption{\textbf{Observed outcomes and added generation in G001.} Left: official successes/executed branches for 12 candidates from 11 training tasks; the two Django 13449 rows are different candidates. Submission is evaluated once, other actions at two seeds. Right: candidate-weighted output-token means over 12 submission evaluations or 24 nonterminal branches, excluding initial generation, prefill, tests, grading, and decision overhead. All repair actions use the frozen 9B model and the common 8,192-token/30-tool/1,200-second limits, with up to 300 additional seconds for a pretest. None of the three initially unsuccessful candidates is repaired. Repeated candidates and seeds are not independent tasks.}
\label{fig:action_pilot}
\end{figure*}

No action repairs any of the three initially unsuccessful states. Continuation and regression feedback preserve all observed submission outcomes, while targeted feedback damages one successful state in one seed. Restart damages 15 of 18 initially successful branches. All 24 restarts reach a budget limit: 16 hit the tool-call cap and eight hit the output-token cap. This characterizes the specified short restart policy, not unrestricted restart quality. Two long-history states also produce context-limited branches. The observed per-state oracle cannot improve on always submitting in this cohort, so action disagreement alone is insufficient evidence for state-dependent control.

The completed G001 allocations consumed 1.805 A100-hours, including startup, idle time, smoke runs, and earlier infrastructure attempts; this is collection cost, not deployment savings.

\paragraph{Frozen failure-enriched diagnostic.}
Before observing any new action outcomes, we selected one originally unsuccessful final candidate from each of 12 previously unbranched training tasks. A fixed hash selected tasks and candidates after a prespecified context-fit check; no development or test task was used. We evaluated submission, continuation, targeted public feedback, and broader public feedback, with two seeds for each nonterminal action. Of 84 frozen records, 83 receive an official verdict. The remaining targeted-feedback branch has a transport read error and retains an unavailable outcome. All 12 submitted patches fail as expected.

\begin{table}[t]
\centering\small
\caption{Failure-enriched F1 diagnostic. Only originally unsuccessful training final states are included. Each successful test arm comprises the same task at two seeds; the one unavailable targeted outcome is excluded from its success denominator. These fractions are conditional feasibility observations, not deployment accuracy.}
\begin{tabular}{@{}lrr@{}}
\toprule
Action & Judged & Successful\\
\midrule
Submit & 12/12 & 0\\
Continue & 24/24 & 0\\
Targeted feedback & 23/24 & 2\\
Broader feedback & 24/24 & 2\\
\bottomrule
\end{tabular}
\label{tab:f1}
\end{table}

The four successful repair executions arise from a single Django task: targeted and broader feedback each repair it at both seeds. No other judged branch repairs another task, and no pair of non-restart actions has replicated opposite preferences across different tasks. The prespecified gate required at least 95\% judged coverage, three distinct tasks with replicated repair, and a replicated action crossover. Coverage is 83/84 and the baseline verdicts match, but the repair and crossover criteria fail. F1 consumed 0.783 additional A100-hours. We therefore stop full final-state branch expansion and action-utility model training under this protocol. The existing development and 60-task test cohorts remain sealed for action selection.

\paragraph{Baselines and attribution.}
The originally planned full comparison would include fixed policies, a CPU gradient-boosted predictor, direct action heads, and current-correctness prediction with empirical action-transition kernels. Those action-utility models were not trained: the feasibility gate stopped the expansion before a policy comparison. The completed evidence is therefore a matched-action diagnostic, not a controller benchmark.

\paragraph{Statistical scope.}
Tasks, rather than branch seeds, are the unit for any future paired interval. G001 covers 11 tasks and F1 is deliberately enriched for failed final states. The two-percentage-point tolerance and 15\% saving target were engineering criteria, not achieved results. One F1 outcome is unavailable, and the 60-task test cohort remains sealed.