实验与推断记录

Experiment Log

(Format follows the research-paper-writing skill, Step 4.6; it is the bridge from results to the paper plan. Numbers: NARRATIVE_REPORT.md v1.1 and the data files it lists. Twelve evaluation arms are final; no reruns.)

Contribution (one sentence)

SFT on 8,137 trajectories of a frontier coding agent transfers the teacher's task-independent habits and input-token footprint within one epoch, while pass@1 gains depend on the starting model (+8.9 pp on a bare base, nothing resolvable on post-trained bases at 731/500 tasks).

Experiments Run

Experiment 1: Dose–response SFT on the dense line (Qwen3.6-27B + v12 ×1/2/3, + v13 ×1)

• Setup: 8,137 v12 trajectories (Claude Opus 4.8 in Claude Code, verified-pass), 1/2/3 epochs = step128/256/381; v13 = 9,880 high-precision + 14,081 by-product trajectories with masked false-success endings, 1 epoch; same hyper-parameters (LR 8e-6, seq 196,608, GBS 64).
• Evaluation: SWE-bench Pro 731 + Verified 500, pass@1, Claude Code 2.1.202, temperature 0.8, paired exact McNemar.
• Result: Pro 50.1 → 49.7 → 52.8 → 51.0 (%), v13 52.3; Verified 68.4 → 67.8 → 68.6 → 71.2, v13 66.2. No baseline-relative pair significant by exact p < 0.05 except v13 vs s381 on Verified (−5.0 pp, p = 0.0046, worse); s256 vs s128 Pro +3.15 pp (p = 0.014) — all to be Holm-corrected.
• Fingerprint: composite JS to teacher 0.0285 → 0.0226 → 0.0215 → 0.0213; Bash-vocabulary JS 0.202 → 0.069 → 0.053 → 0.050; thinking-position JS 0.0162 → 0.0085–0.0110.

Experiment 2: MoE line — bare base vs post-trained base (35B-A3B)

• Qwen3.5-35B-A3B-Base + v12 ×2: Pro 35.4 → 44.3 % (+8.89 pp, p = 4.5e-7), Verified 52.0 → 55.4 (+3.4 pp, p = 0.13). Qwen3.6-35B-A3B native + v12 ×3: Pro 46.1 → 48.8 (+2.74 pp, p = 0.075), Verified 63.0 → 63.0 (p = 1.0). Bash JS 0.278 → 0.057 (bare), 0.195 → 0.061 (post-trained).
• Temperature-0.6 reruns of both Qwen3.6-A3B arms on Pro: 97 and 83 of 731 tasks flip, net +0.96 / +0.68 pp (p = 0.54 / 0.66).

Experiment 3: Vendor next generation as a natural comparison (Qwen3.8-27B native)

• Pro 55.7 %, Verified 74.2 %; vs Qwen3.6-27B +5.61 pp (p = 1.1e-4) / +5.80 pp (p = 0.0022). Cumulative input 4.41 M tokens/task (2.9× the distilled 27B arm). Fingerprint: Bash JS 0.061 and thinking-position JS 0.010 inside the distilled band; tool-mix/bigram JS 0.031/0.056 (farther than native 27B); recovery@3 0.338; write-without-read 0.177; whole-file Write 22 % of writes.

Experiment 4: Same-task anchor (88 training-set tasks, n27 vs s256, ≈3 rollouts each)

• Read-file-set Jaccard to teacher 0.60 vs 0.44 (sign test 66:15); tool-mix L1 0.219 vs 0.471 (77:11); edit-set Jaccard 0.75 vs 0.65 (44:24, p = 0.021); solve rate 61.6 % vs 55.3 % (24:13, p ≈ 0.10; medians 2/3 each). Rollouts: 263 vs 296 trajectories (2–4 per task).

Experiment 5: Unified fingerprint extraction (46,542 trajectories) and language strata

• One extractor over teacher trajectory.json and student session jsonl; three gates green; teacher-internal noise floor 0.002–0.003. Within-language Bash JS (n27 → s256): Python 0.341 → 0.133, Go 0.217 → 0.066, JS/TS 0.237 → 0.105. Teacher Python-only reference (n = 574): composite n27 0.0267 → s256 0.0195 → s381 0.0178; on Verified composite does not converge (0.0369 → 0.0465) while Bash does (0.136 → 0.071).

Experiment 6: Error handling, thinking, wording (pooled rates)

• recovery@3: teacher 0.603; n27 0.299; s128/s256/s381/v13 0.605/0.580/0.625/0.599; a3b_q36 0.329 → a3b_v12 0.593; q38 0.338. Post-error thinking ratio: teacher 0.34, n27 1.04, s256 0.47 (denominator-confounded; absolute values pending). Thinking share: teacher 0.893, corpus 0.8805, students 0.925–0.946, q38 0.952. Failure-ending strong success claims (Pro): n27 66.3 % → s128 52.9 → s256 49.1 → s381 48.9 → v13 44.1; teacher 44.6; q38 24.1; audit of 240 endings: regex precision 69/70; 0 of ~110 regex-negative endings admit failure.

Experiment 7: Cost decomposition

• Cumulative input per task (Pro medians): 27B native 3.27 M → +v12×2 1.52 M (steps −32 %, per-step context −29 %); MoE 3.27 M → ≈2.1 M; q38 4.41 M. Whole-file reads 7,293 → 2,578 (−65 %) at constant size per read; truncation pipes 47 → 69 %; output tokens +16 % (27B) / +49 % (MoE); matched-pair wall-clock −1 %.

Experiment 8: Leak audit and decontamination

• Golden-patch audit with negative controls (network / local git history / package manager): conclusions unchanged on clean subsets; git-history reading 140 → 92 trajectories after SFT (A3B line). Instance-level train/eval overlap: 0 title matches, max description Jaccard 0.21, 0 commit-SHA hits; Verified repos 0 overlap; teleport repo shares 76/731 Pro tasks with 1,622 teacher trajectories (teleport-excluded tables in analysis/stats/).

Figures

• Fig 1 (hero, two panels): epoch × Bash-JS (log) with teacher-noise band and q38 marker / Pro pass@1 with MDE band; cumulative input × pass@1 plane with three arrows (corpus, corpus-on-bare-base, generation). Source: figures/fig1.tex (pgfplots).
• Fig 2: same-task anchor per-metric sign tests + scatter. Fig 3: "what moved toward the teacher" multi-row figure (Bash heads, recovery, thinking position/quantity, wording). Fig 4: cost waterfall (steps × per-step) and observation sources. Fig 5: flip rates and detectable effect vs rollouts. Table 1: 12 arms × 2 pools with paired p (Holm). Table 2: failure-ending wording with audit counts.

Failed Experiments (document for honesty)

• v13 (masked false-success endings) did not change honesty (0 admissions either way) and cost −5.0 pp on Verified vs v12×3; the mask covered only strong-form wording.
• "Channel narrowing" as a general claim failed the judge: the bare MoE base moves in the opposite direction on sub-agent use; the teacher is not at 0 % (0.28 % dispatch calls), and teacher side-chain capture is incomplete.
• Temperature-only reruns showed 11–13 % task flips; "+18 tasks" pass@1 gains dissolved under rollout×3 re-estimation.

Open Questions

• Does the habit-layer convergence hold on a held-out same-pool anchor (373 never-trained tasks with teacher trajectories)? (must run)
• Does the corpus' token-footprint reduction graft onto Qwen3.8-27B without losing its +5.6 pp? (should run)
• Do the habits transfer across harnesses (mini-SWE-agent)? (nice to have)
• Human re-annotation of 240 + 68 endings (two annotators, κ); absolute post-error thinking; Holm-corrected test family; MDE on 731 tasks; same-weights and split-half noise floors for the four distances (analysis agents running 2026-09-02).