P论文工作台最新版本 ↗
完整资料 / TEX_A_APPENDIX

A_appendix · 完整稿件源码

paper-rl/paper/sections/A_appendix.tex

manuscript观察/版本:2026-10-0121,114 bytes下载完整原文直接读取
\section{Implementation and Replay Evidence}
\label{app:validation}

\paragraph{Factual reconstruction.}
The original development pilot contains 40 tasks with eight planned rollouts per task. An index-only rule selects repetitions 0, 3, and 6 of every task for cross-machine replay, giving 120 fixed cases. Every expected test's reported success/failure label and the task identifier are compared. All 120 cases match; unavailable cases are not replaced. Factual replay and intervention validation are separate checks.

\paragraph{Task controls.}
Each of the 149 admitted tasks has a gold-patch control with resolved verdict one and an unchanged-tree control with verdict zero under a matching profile: 298 controls. These check the configured expected-test score, not every unscored suite test. One excluded candidate's official lists omit the issue-specific failing test, so its unchanged-tree fixture scores as resolved locally; the official command-line evaluator separately filters empty submissions. The original 500-task evaluation membership and labels are retained by the protocol. Two tasks have common resource corrections: two CPUs for a required two-core test, and one numerical-library thread under an existing one-CPU quota.

\paragraph{Evaluation-profile controls.}
Two additional reference pairs illustrate different measurement limits. For \texttt{psf\_\_requests-1724}, both the gold patch and unchanged tree pass all 85 fixed expected tests and receive native score one; the sole suite failure is outside those lists. This pair does not distinguish the patches under the recorded profile. For \texttt{psf\_\_requests-2317}, the gold patch passes all eight fail-to-pass tests, but an expected pass-to-pass test raises a connection-reset error during an external HTTP request, producing native score zero. The unchanged tree also fails the targeted encoded-method test. Each of these four reference controls has one recorded execution. These records establish neither network reliability nor a repaired evaluation environment; their original scores and the fixed benchmark membership are retained. The logs, expected-test observations, and source hashes are indexed in \path{requests_control_observations_20260927.json}.

\paragraph{Fixed lists and empty submissions.}
Each gold and unchanged-tree control in the historical and common-profile pairs for \texttt{django\_\_django-10097} reports native resolution one. In both common-profile controls, all 1,870 fixed native identifiers are recorded as \texttt{PASSED}; six full \texttt{URLValidator} test cases pass for gold and fail for the unchanged tree, and none belongs to the grading lists. These are reference test-report scores: the final evaluator's separate empty-prediction rule remains zero. The historical pair lacks original observation artifacts, so its test reports remain unresolved under the strict evidence check. The task and grading lists remain fixed; the controls and source comparisons are indexed in \path{django10097_control_observations_20260928.json}.

\paragraph{Test phases and native aliases.}
A recorded call phase does not establish execution of the intended assertions. In a gold/empty reference pair for \texttt{scikit-learn\_\_scikit-learn-15100}, whitespace truncation merges three full parameterized test identifiers into one fixed native identifier. The \texttt{CountVectorizer} and \texttt{TfidfVectorizer} instances pass, while the \texttt{HashingVectorizer} instance produces a call-phase \texttt{XFAIL}. The pinned test source calls \texttt{pytest.xfail} for that estimator before the subsequent assertion block. Its reason mentions PyPy, but the branch does not check the runtime platform. The native gold score remains one; its strict admission remains unresolved. Complete identifiers, marker declarations, phase reports, and source bindings are indexed in \path{sklearn15100_xfail_attribution_20260928.json}.

\paragraph{Observed exceptions and unresolved evidence.}
An audit of nine selected SymPy task pairs reproduces all 18 saved native grading results. Each unchanged-tree control has one expected fail-to-pass test-body exception, every expected pass-to-pass test passes, and no expected identifier is missing. The native zeros remain unresolved under the strict admission rule. Qualified \texttt{file:test} entries marked \texttt{FAILED} alias the same exceptions recorded as bare-name \texttt{ERROR}; they are not independent failed-test witnesses. The same two additional errors outside the fixed lists occur in each SymPy19637 control and trace to Python AST compatibility. These reference diagnoses do not classify failures from future model patches. Exact evidence and unchanged decisions are indexed in \path{batch17_18_sympy_attribution_20260928.json}.

\paragraph{Implementation checks and records.}
The 20 reviewed CPU checks comprise four integration probes, eight readiness tests, four terminal-focus algebra tests, and four generator/trainer-hook tests. They cover alignment, observation/padding masks, terminal-return preservation, and zero-credit equivalence through GRPO, prompt-mean, and DPPO. Separate coefficient-array and synthetic backward checks cover nonzero M/B tensors. None of these is a distributed GPU optimizer test. Two original controls lack grading reports; the archived control and repair configurations reuse their project names. The original records are retained, and replacement controls use unique identities.

The reproducibility records are \path{reconstruction120_20260927.json}, \path{training_profile_validation_20260927.json}, \path{clean_pilot_validation_20260927.json}, and \path{pilot_mechanism_strata_20260927.json}, with paths and hashes of their raw inputs. Service-repeat records preserve every issued artifact. The empirical 95th percentile is sorted element $\lfloor0.95(N-1)\rfloor$ with zero-based indexing. The original service measurement exceeds the development targets of mean 180 seconds and 95th percentile 360 seconds; the deployment repeat is a separate operational measurement, not a retrospective replacement.

\paragraph{Grading failures and training recovery.}
\label{app:grading_recovery}
A later audit found numeric zeros without native grading reports in an O training run. A frozen consumption snapshot links one such trial to training call five and 18 to call 40. The run was stopped after 40 completed calls and excluded from the planned comparison; its checkpoints were not used to initialize the replacement. Successor training uses the same base and task membership, with an offline copy of the pinned native grading functions. It reproduces the reports and rewards of 265 eligible archived reference records, and a new \texttt{pydata\_\_xarray-4966} gold/empty pair receives one/zero with complete expected-test observations. These checks do not repeat all 298 admission controls above. Numerical verifier rewards require matching report, observation, completion, and source records. Missing evidence remains unjudged under bounded retry and masking rules. Missing token metadata or an exception caught while processing an established terminal reward masks the trajectory without another generation. Caught auxiliary-credit exceptions retain terminal learning. This changes feedback missingness and cost, not the native scoring formula. Records and the earlier failures are retained in \path{O42_recovery_validation_20260927.json} and its bound evidence.

\paragraph{CPU quota and skipped expected tests.}
For \texttt{pylint-dev\_\_pylint-6528}, a one-CPU profile skips four expected tests through a two-core eligibility hook. Same-image probes return effective CPU counts of one and two under the corresponding quotas. A separate profile changes only the quota to two CPUs. Its single gold/empty pair scores one/zero, with all 175 fixed test identifiers observed in each execution. Two failures outside the fixed lists remain in each execution. The original pair and grading lists are preserved; the candidate has not been adopted as a common evaluation profile. This is an environment control, not a policy comparison.

\section{Algorithm Conventions}
\label{app:algorithm}

\paragraph{Snapshots and eligibility.}
An initial snapshot precedes generation. After each assistant turn's sequential shell actions finish, a private Git index stages tracked changes and non-ignored untracked files within the repository using \texttt{git add -A}; it does not overwrite the agent's index. A Git tree records the resulting files, and a no-rename diff against the previous tree gives the changed paths. A format-error turn executes no action and appends an unchanged snapshot, ending a contiguous editing block. Attribution is to this observation boundary, not to individual processes that wrote files.

All paths touched within a block are considered, including paths changed and restored within it. Later changes invalidate the block even if those changes are subsequently undone. Test-path exclusion is the union of the following case-sensitive regular expressions, applied to repository-relative paths:
\begin{quote}\small
\verb!(^|/)(tests?|testing)(/|_)!;\quad
\verb!(^|/)test_[^/]*\.py$!;\\
\verb!_tests?\.py$!;\quad
\verb!(^|/)conftest\.py$!.
\end{quote}
All edit blocks receive chronological zero-based indices before eligibility filtering, and eligible blocks retain those indices. The selector minimizes the integer SHA-256 digest of \texttt{key|index}; \texttt{key} is \texttt{instance\_id|rep|cache\_salt}. Training uses the task-alias identity and sampling weight-version salt; the development pilot uses the task identity and literal pilot salt. A hash tie retains the first block. Selection does not inspect a test verdict. The counterfactual patch is a binary diff of the constructed intervention tree against the initial baseline.

\paragraph{Delivery procedure.}
For each generated trajectory: (1) record snapshots and generated-turn spans; (2) form and select an eligible block; (3) restore its touched paths in the final tree; (4) reject a failed snapshot, construction error, unchanged intervention, format error, or incomplete receiving span; (5) for B/M, read the original utility and issue one intervention trial; (6) compute $D,s,L,m$ and the corresponding coefficient array, or record an unavailable query; (7) apply the common trajectory masks and add surviving coefficients after group centering. E applies the same structural and complete-span checks, omits utility parsing and querying, and uses the terminal advantage at the trainer. Original and intervention utilities use the same fixed expected lists; unavailable utilities never become synthetic zero measurements.

\paragraph{Common trajectory masks.}
A failed trajectory without usable token metadata returns a zero loss mask. A prompt group containing a final error is wholly masked; two or more unjudged agent timeouts also mask the group, while a single such timeout masks only that member. Judged timeouts with complete metadata retain their actual reward. Context-truncated generated tokens remain trainable in this configuration, but incomplete selected-turn spans cannot receive auxiliary credit. Observation and padding tokens are always excluded. Format errors retain the common terminal penalty but receive no auxiliary term.

The successor training adapter uses at most two attempts for retryable, still-unjudged errors. A missing reward file remains unjudged even when verifier stdout exists; the earlier zero fallback was removed after the recovery audit. Unavailable original utility also prevents a CF query. Missing token metadata or exceptions caught in terminal-reward processing mask the trajectory without another generation; caught auxiliary-credit exceptions retain terminal learning. These training masks remain separate from the final evaluation protocol's explicit failure attribution.

\paragraph{Policy surrogate and budgets.}
Let $p_{it}=\pi_\theta(a_{it}\mid h_{it})$ and $b_{it}$ be the recorded behavior-policy probability. The implementation uses $\rho_{it}=\exp(\operatorname{clip}(\log p_{it}-\log b_{it},-20,20))$. For $z=\widetilde A_{it}$, the binary-TV mask in \cref{eq:loss} is
\[
q_{it}=\begin{cases}
0,&z>0\text{ and }p_{it}-b_{it}>0.15,\\
0,&z<0\text{ and }b_{it}-p_{it}>0.15,\\
1,&\text{otherwise}.
\end{cases}
\]
The mask is detached from differentiation. Advantages are divided by $N_g\max(1,T_g)$ before the token losses are summed. No additional importance-sampling correction is enabled. Empty groups contribute zero. The base checkpoint is the frozen Qwen3.5-9B snapshot identified by revision prefix \texttt{c2022362}; exact input identities are bound in the run records. Agent and verifier task timeouts are each 3,000 seconds before the common multiplier of two. There is no separate turn-count limit; context and wall-clock budgets apply. Commands time out after 30 seconds, and observations are truncated to 3,000 characters.

\section{Statistical and Cost Protocol}
\label{app:statistics}

The analysis specification was fixed before policy evaluation. Let $y_{kir}\in\{0,1\}$ be task $i$'s resolution on repetition $r$ for training seed $k$. For arm $a$, $N$ tasks, $n$ repetitions, and $K$ training seeds, define
\[
 \widehat p_a=\frac1K\sum_{k=1}^{K}\frac1N\sum_{i=1}^{N}
 \frac1n\sum_{r=1}^{n}y^a_{kir},
 \qquad \widehat\Delta_{M-B}=\widehat p_M-\widehat p_B.
\]
The primary population is the designated holdout 100 with $n=16$ and $K=3$ paired training seeds. M--O is secondary; M42--E42 has one training seed. The 500-task $n=2$ cohort is independent and supplementary; its two attempts are not pooled into the primary cohort or selected for favorable scores.

The appendix will report approximate two-sided 95\% bootstrap intervals from 10,000 draws with random seed 20260927. Each draw independently resamples training seeds and tasks with replacement, using the same indices across compared arms and retaining all repetitions within a sampled cell. E42 comparisons resample tasks only. Three training seeds provide limited information about training randomness, so intervals accompany individual seed differences and effect sizes. Supplementary partitions are not alternative primary populations, and repeated trajectories are not independent training runs.

\paragraph{Attempt identity and missingness.}
Execution binds checkpoint, tokenizer, service, task profile, grading lists, and runner hashes to immutable manifests. Each logical task/rep slot accepts the first valid report, regardless of its score. An explicit verifier-infrastructure failure can receive at most one re-verification of the identical patch; a generation-infrastructure failure before a complete trajectory can receive at most one replacement generation. All attempts and reasons are retained. Valid failures, model-caused budget exhaustion, and model-broken installation are not retried for a better outcome. If a model-caused termination still has a valid report, that report is used; without a judgeable final artifact an established model failure is zero. Ambiguous attribution remains unresolved.

With $S$ observed successes, $J$ observed outcomes and $H$ planned slots, unresolved slots yield bounds $[S/H,(S+H-J)/H]$, not an imputed point score or a reduced denominator. Difference bounds subtract opposite arm endpoints. A comparison with unresolved outcomes is reported as incomplete, without a completed-effect point estimate or significance claim. Logical slot manifests have been fixed; checkpoint loading/precision calibration, full evaluation profiles, and the strict attempt runner remain execution prerequisites.

\paragraph{Cost.}
Accounting includes allocated GPU-hours, generated tokens, original and CF verifier trials, service and queue time, failed attempts, and unused prefetches. A single extra logical query bounds query count, not total resource cost. Historical runs with invalid or incomparable feedback are excluded from policy comparisons.

\section{Origin of the Intervention Signal}
\label{app:signal-origin}

For the same 244 usable development queries, write $D=\Delta_Y+\Delta_F+\Delta_G$, where
\[
\Delta_Y=Y(P)-Y(P^{-e}),\qquad
\Delta_F=0.25\,[F(P)-F(P^{-e})],\qquad
\Delta_G=-0.25\,[G(P)-G(P^{-e})].
\]
The following table accounts for all 17 nonzero contrasts among the 111 queried failed originals. The breakdown is unchanged in the admitted 39-task stratum.
\begin{center}
\small
\begin{tabular}{lrrrr}
\toprule
Contrast on a failed original & Count & $\Delta_Y$ & $\Delta_F$ & $\Delta_G$\\
\midrule
Positive, F2P component & 6 & 0 & $0.125$ or $0.25$ & 0\\
Positive, regression component & 6 & 0 & 0 & $0.25$\\
Negative, F2P component & 1 & 0 & $-0.25$ & 0\\
Negative, regression component & 3 & 0 & 0 & $-0.25$\\
Intervention resolves the task & 1 & $-1$ & $-0.25$ & 0\\
\bottomrule
\end{tabular}
\end{center}

For six positive contrasts, the issue-test component is unchanged and the regression indicator supplies the entire utility difference. In one negative case, repetition 6 of \texttt{sympy\_\_sympy-15875}, restoring the selected \path{sympy/core/add.py} changes $(Y,F,G)$ from $(0,0,0)$ to $(1,1,0)$, giving $D=-1.25$. All other final files remain fixed during that intervention. The recorded decomposition and per-query input hashes are provided in \path{pilot_credit_origin_20260927.json}.

\section{Observed Coefficient Geometry and Example}
\label{app:geometry}

The following summaries use all 92 usable nonzero development queries and the lower empirical quantile convention above. Magnitudes refer to each arm's receiving tokens before adding the terminal advantage or applying DPPO. The cap removes magnitude information from 83 of these cases; this is a property of the measured setting, not evidence that saturation improves learning.
\begin{center}
\small
\begin{tabular}{lrrr}
\toprule
Quantity & 5th percentile & Median & 95th percentile\\
\midrule
Selected generated-token fraction $m/L$ & 0.00504 & 0.02100 & 0.04823\\
Shared shrinkage $s$ & 0.04035 & 0.18067 & 1.00000\\
M receiving coefficient magnitude & 0.77242 & 1.00000 & 1.00000\\
B receiving coefficient magnitude & 0.00504 & 0.02077 & 0.04823\\
\bottomrule
\end{tabular}
\end{center}

\begin{samepage}
For a concrete recorded example, repetition 7 of \texttt{django\_\_django-14007} selects zero-based turn 105 and path \path{django/db/models/query.py}. The final patch changes only \path{django/db/models/base.py}; the selected turn restored the former file after earlier agent edits. Reinstating its pre-block contents reintroduces an indentation error, so the intervention fails before the expected tests can pass. The original has $(Y,F,G)=(1,1,0)$; the intervention has $(0,0,1)$, giving $D=1.5$. Here $L=21{,}162$, $m=186$, and $s=0.05860$. M receives coefficient $1$ on the selected tokens and B approximately $0.00879$ on all generated tokens; each has total signed coefficient mass 186. This is a traceable self-repair example, not an estimate of attribution accuracy.
\par
\end{samepage}

\section{Configuration Companion}
\label{app:configuration}

The accompanying \path{configuration-v1.zip} preserves the earlier training configuration, task/split manifests, 149 admission-control task profiles, agent template, local tokenizer/chat-template metadata, and content hashes. It predates the grading recovery in \cref{app:grading_recovery}. Host addresses and workspace paths are replaced by explicit placeholders; weight hashes retain their original identities. It is a configuration companion, not a complete executable runtime release. The recovered source/runtime package and final evaluation artifacts must accompany a submission.

The four local weight shards match the public Qwen/Qwen3.5-9B revision \path{c202236235762e1c871ad0ccb60c8ee5ba337b9a}. Local tokenizer and template files have their own recorded hashes. Training uses the language-model-only parameters with LoRA rank zero, BF16 parameters, FP32 gradient reduction, tensor parallelism four, and one update epoch per batch. The configured microbatch is one sequence per GPU; actual packing/microbatch counts are recorded per update. Optimizer settings are betas $(0.9,0.98)$, weight decay $0.01$, gradient-norm cap $1$, and constant learning rate $10^{-6}$ without warmup. The 14 allocated GPUs are A100-SXM4-80GB; each of the two shared hosts has two 32-core Xeon Platinum 8350C CPUs. These shared resources do not imply dedicated verifier CPU time.

The corrected task identities are \texttt{pylint-dev\_\_pylint-7277} (two CPUs) and \texttt{scikit-learn\_\_scikit-learn-25232} (one BLAS/OpenMP thread). The excluded training candidate is \texttt{django\_\_django-7530}. Membership lists and original grading configurations in the companion make these changes explicit.

\section{Assisted Development}
Language-model tools assisted implementation, code review, and manuscript drafting. Experimental records and numerical summaries are retained separately from the manuscript. Implementation reviews are same-family, provisional checks; empirical conclusions depend on executable measurements.

公开内容 SHA-256 8ac3f4391ebe659b2a91869adf36d1eab7bd148673a6ee2f48615fabb2d13457

返回资料与接手