2_related_work · 完整稿件源码

\section{Related Work}
\label{sec:related}

\paragraph{Counterfactual utility and executable reduction.}
Difference-reward constructions precede language-model agents: wonderful-life utility subtracts utility after clamping part of a realized system \citep{tumer2002collective}. COMA instead marginalizes one agent's action through a centralized critic while holding others fixed \citep{foerster2018counterfactual}. In software, delta debugging repeatedly tests reduced inputs or differences to isolate failure-inducing elements \citep{zeller2002simplifying}. Our adaptation uses one selected final-artifact intervention to produce a training coefficient and maps it to the recorded producing turns. It does not perform iterative minimization, estimate an action-marginal baseline, or inherit guarantees from effect-set clamping.

\paragraph{Execution feedback and multi-turn learning.}
SWE-bench evaluates repository changes against task-specific tests \citep{jimenez2024swebench}. Such executable feedback supplies an outcome for a complete repair, whereas optimization must act on the sequence of decisions that produced it. Group-based policy optimization offers a simple outcome-based starting point: the reward of a sampled solution is compared with rewards from the same prompt group \citep{shao2024deepseekmath}. This establishes a training signal for the sampled response, but leaves a separate question about allocation within a long interaction. SWEET-RL studies collaborative, multi-turn tasks and trains a critic that can use additional information available during training \citep{zhou2025sweet}. Its local feedback is learned from that information. Counterfactual edit credit instead obtains a particular local signal by executing a modified completed artifact. The distinction concerns how the feedback is constructed, rather than whether multi-turn agents can already receive intermediate credit.

\paragraph{Local feedback for generated software.}
BoostAPR combines execution-verified demonstrations with sequence-level and line-level reward models, then uses the latter to allocate repair feedback to edit regions \citep{li2026boostapr}. This makes execution-derived local credit a direct precedent for software training. RCCA connects functional rubric feedback for interactive applications to responsible source spans and generated tokens \citep{jin2026rcca}. Both motivate a distinction between assessing an artifact and assigning optimization weight inside its generation. Our construction uses a different assessment: the difference between the final repository and an artifact with selected file effects removed. Its receiving region is also different. We allocate to the complete generating turns of an edit block, rather than asserting a precise mapping from a diagnostic or reward-model prediction to individual code tokens. This choice keeps the intervention and its recorded producer aligned at the granularity supported by the trace.

\paragraph{Separating credit magnitude and allocation.}
FACTOR explicitly separates action-credit estimation from token allocation. Its action credits use checkpoint-calibrated temporal differences, while feedback-conditioned likelihood gaps distribute those credits within actions \citep{ma2026factor}. CoRT contrasts the likelihood of a realized response under rubric-conditioned and criteria-free prompts, then uses bounded, normalized token weights to redistribute the response advantage \citep{zhang2026cort}. The conserved quantity and the loss reduction matter. FACTOR normalizes within each action and couples that normalization to an action-mean surrogate. Its checkpoint-calibrated action credits telescope to the trajectory advantage, with sparse continuation estimates used for calibration. Our equality instead concerns the trajectory-level sum of an additive coefficient over generated positions. We retain prompt-mean reduction and add the terminal advantage before DPPO masking, whereas FACTOR preserves each action's own TD credit through within-action normalization and an action-mean reduction. These constructions therefore control different quantities even though both conserve a form of credit. The broadcast control here uses the same removal utility and cap shrinkage as localization on a shared trace, so its comparison is tied to this exact estimator and reduction. Conservation alone cannot transfer an optimization interpretation from one construction to the other.

These works establish that conserving or redistributing a scalar credit is not itself a new contribution. Our comparison uses one artifact-derived \emph{additive} signal with two receiving masks. Because the signal is added to the original group advantage, it can remain nonzero when the terminal group advantage vanishes. The shared mass constraint controls one aspect of this comparison; it leaves concentration, sign interactions with terminal advantages, and subsequent trust-region masking visible in the analysis.

\paragraph{Continuation-based and reversible execution.}
VinePPO estimates intermediate values from Monte Carlo continuations, providing a route to credit assignment without a learned value network \citep{pmlr-v267-kazemnejad25a}. Shepherd represents agent execution as a reversible trace and supports fork-based operations; its training application uses forks to obtain alternative rollout feedback \citep{yu2026shepherd}. These approaches can inspect what follows an intermediate state under further execution. Our intervention holds the other final files fixed and changes only the selected block's surviving file effects. It therefore measures a conditional artifact contrast, not the expected outcome of letting the agent continue after omitting an action. The absence of an additional policy continuation is a property of this construction. This ties the feedback to the final file effects and their recorded producing turns, with verification cost accounted for separately.