Paper Plan (v1, after cross-review 2026-09-02)
Title (working, reviewer's proposal): What Agentic SFT Buys: Rapid Behavioral Convergence, Base-Conditional Capability Gains Alternatives (rejected by the reviewer: too long / "cheap" implies money / "clones" overstates): *Curated Frontier-Agent Trajectories Transfer Habits and Token Footprint in One Epoch* · *Habits Are Cheap, Capability Is Not* · *Agentic SFT Clones the Process, Not (Necessarily) the Capability* One-sentence contribution: A curated corpus of 8,137 frontier-agent (Claude Opus 4.8 / Claude Code) trajectories, built by a rule set we describe and measure, moves open students' command vocabulary, error recovery, thinking rhythm and input-token footprint toward the teacher — most of the dense model's convergence after one epoch — while pass@1 response depends on the starting model: +8.9 pp on a bare base (Pro), and −2.2 to +2.8 pp against post-trained baselines with no comparison resolved at this evaluation scale. Positioning sentence (from the novelty gate, to use verbatim in §1): Prior work shows that agent behaviors can be fingerprinted and made more teacher-like; we show where that convergence saturates, what operational habits and input-token footprint it transfers, and why it can be sharply dissociated from paired solve-rate gains on post-trained coding models. Venue: COLING 2027 (Macau, 9–14 May 2027) via ACL Rolling Review, October 2026 cycle (deadline 2026-10-12 AoE); ACL 2027 / Findings as the fallback commitment in the same cycle. (Changed 2026-09-02 from ACL/EMNLP; COLING 2026 had already taken place.) Type: empirical analysis + data-curation description. Date: 2026-09-02. Page budget: 8 pages main body (ACL long) inclusive of floats: front matter 0.5, §1 0.8, §2 0.45, §3 1.4, §4 0.65, §5 1.8, §6 1.15, §7 0.3, §8 0.2 (= 7.25, buffer 0.75); Limitations mandatory and uncounted; appendix unlimited. Body floats: Fig 1, Fig 2/Table 2 (funnel + rules), Table 1, Fig 3 only. Section count: 8. Presentation rules (user): no confidence intervals and no significance tables in the body; scores are pass@1 with pp differences, statistics go to an appendix; the innovation lines are (i) behavioural analysis of what transfers and (ii) the SFT data-curation pipeline and what its rules change; the twelve evaluation arms are final, infrastructure-caused reruns are not mentioned.
Claims–Evidence Matrix
| # | Claim (planned wording; H/S rows follow the result-to-claim accepted wordings, D rows are provisional — added after the gate) | Evidence | Status (gate) | Section |
|---|---|---|---|---|
| D1 | The corpus is a curated subset built by an explicit rule set: hidden-test verification + vendor quality audit → 12,257 → P0 drop rules remove 28.5 % (raw-thinking coverage < 0.98, answer-source fetches, content leaks, sandbox escapes, multi-user turns) → same-task cap → 8,137 train / 425 val; message-level loss masks on 5,528 rows (harness-rejected calls, fabricated cwd, assertion removal, silencing) | analysis/data/data_pipeline.md funnel + rule tables; train/data/swepro-v12-*/raw/build_report.json | supported (descriptive; counts recomputed) | §3 |
| D2 | The composite v13 intervention (relaxed thinking floor, 14,081 by-product trajectories = 7,503 passes + 6,578 masked failures, no task cap, one epoch) vs v12×3: Pro +1.3 pp, Verified −5.0 pp; fingerprint and footprint indistinguishable from the v12 arms; strong-form failure wording 48.9 → 44.1 %; 0 admissions either way. No component effect is identified (filters, composition, task cap and epochs change together) | report_excerpt tests v13 vs s381; fp_summary overclaim; fp_pooled; think_overshoot | provisional (descriptive; confounded) | §3, §6 |
| D3 | Students' thinking share (0.925–0.946) exceeds both the corpus (0.881) and the pooled teacher reference (0.893); the design does not identify the cause | think_overshoot.json; fp_pooled | provisional (descriptive) | §5 |
| D4 | Habits present in the corpus move toward or reach the teacher reference on Pro (Bash head-word shares, truncation pipes, low sub-agent dispatch), including an unwanted one — write-without-read (A3B line 72/23 → 138/133 at both temperatures) — while the Edit-parameter hallucination of the bare base is *not* corpus content (0 occurrences in 474,417 calls); these define candidate curation rules, none of which has been ablated | A3B report §7; fp_pooled; corpus scan | provisional (descriptive) | §3, §5 |
| H1 | One v12 epoch accounts for most of the observed dense-model convergence in Bash vocabulary and tool-mix/bigram JS on Pro (bash JS 0.202 → 0.069 → 0.053 → 0.050; composite 0.0285 → 0.0226 → 0.0215 → 0.0213); epoch 2 adds only a Bash-vocabulary step, epoch 3 nothing above the split-half sampling floor; the MoE line converges between its two available points and the strata hold within Python/Go/JS-TS; on Verified only Bash vocabulary and thinking position converge. "Task-independent" is claimed only if the held-out anchor lands | fp_summary; analysis/fp (differences vs floors; Hellinger/TV) | partial → wording adopted; held-out anchor is the claim gate | §5 |
| H2 | Fingerprint proximity changes substantially whether or not pass@1 changes: bare-A3B SFT improves Pro by +8.89 pp but is inconclusive on Verified (+3.4 pp), while no baseline-relative post-trained comparison is resolved (−2.2 to +2.8 pp; Pro-minus-teleport s256 +2.6, s381 −0.2) and several-point effects remain possible; a vendor generation jump gives +5.6 / +5.8 pp at 2.9× cumulative input. Interpretation (softened): on one bare A3B base the result is consistent with the corpus supplying part of the agentic post-training transition; the design does not isolate a general base→instruct mechanism | report_excerpt; analysis/stats (Holm, teleport slice); three-line report | partial → wording adopted | §6 |
| H3 | Input-token footprint is part of the fingerprint: −53 % cumulative input (steps −32 % × per-step context −29 %), via whole-file reads −65 % at constant size (74 % of the dense-line drop, 78 % on the MoE line) and truncation pipes 47 → 68 % (MoE) / 43 → 65 % (dense); identical across checkpoints whose scores disagree; output tokens +16–49 %; matched wall-clock −1 %; token ≠ bill | A3B/27B report pages; report_excerpt cost fields; fp_pooled | partial → wording adopted | §6 |
| H4 | One-rollout paired pass@1 at 731/500 tasks does not resolve the 0–3 pp effects at stake (median detectable effect ≈ 4 pp Pro / 5 pp Verified); task outcomes are temperature-sensitive (11–13 % of tasks flip between temperature 0.8 and 0.6 samples of the same weights — a temperature-perturbation reference, not a test–retest reliability); the SUCCESS_CLAIM regex measures strong completion phrasing, not honesty: 0 of 110 audited regex-negative failure endings admit failure (model-annotated by two models; human pass pending) | analysis/stats; analysis/annot; overclaim_annotation | partial → wording adopted | §4 (two sentences) + §7 + App. A |
| S1 | On 88 trained tasks the student reads fewer files but the teacher's files (read-set precision 0.50 → 0.80, Jaccard 0.44 → 0.60; negative-control Jaccard 0.001), while the solve-rate difference (+6 pp) is imprecise; imitation gain does not predict solve gain (ρ = −0.12) | analysis/fp/anchor_diag; p8_train100 | partial → wording adopted | §5 |
| S2 | On Pro, SFT consistently raises three-call protocol-error recovery toward the teacher in both student families (0.30 → 0.58–0.63; teacher 0.60; vendor next-gen 0.34); Verified is heterogeneous (s128 0.45 → 0.27); the post-error thinking pattern (less than on ordinary steps) is copied while absolute post-error thinking rises — no general "error-handling bundle" claim | fp_pooled; analysis/fp §e | partial → wording adopted | §5 |
| S3 | Across both pools SFT raises thinking-character share above the pooled teacher reference; thinking-position JS generally moves teacherward (0.016 → 0.009 on Pro) except for the post-trained MoE comparison on Verified (0.014 → 0.018) | think_overshoot; fp_summary | partial → wording adopted | §5 |
| S4 | The vendor's next-generation model matches distilled arms on Bash vocabulary, thinking position and truncation pipes, not on tool composition, reading habits or error recovery (descriptive, n = 1) | fp_summary; fp_pooled | yes | §5 |
| S5 | Channel use changes in a base-dependent way (sub-agents down for post-trained bases, up for the bare base); no general narrowing claim; git-history reading −34 % on one line | fp_summary mix; A3B leak section | no → one descriptive sentence | §5 (one sentence), App. |
Structure (8 pages)
§0 Abstract (≈200 words)
- What we achieve: a curated frontier-agent corpus and a behavioural-fingerprint analysis showing what agentic SFT transfers — habits and token footprint within one epoch — and what it does not — capability on already post-trained bases.
- Why it matters / is hard: trajectory SFT is the dominant recipe for open coding agents, but scores alone cannot tell "transfer failed" from "transfer worked but bought nothing" at 731/500 tasks.
- How: 12 arms (dense 27B ×1/2/3 epochs and a by-product variant; MoE from bare and post-trained bases; vendor next generation) in one harness; one extractor over 46,542 trajectories; a same-task anchor; audited endings; a documented curation pipeline.
- Evidence (reviewer's wording): one epoch produced most of the observed dense-model behavioral convergence; SFT cut median input by up to 53 %, improved one bare base by +8.9 pp on Pro (+3.4 pp, inconclusive, on Verified), and yielded −2.2 to +2.8 pp against post-trained baselines, with no exact baseline comparison resolved; a generation jump gave +5.6–5.8 pp at 2.9× cost; 0/110 audited failure endings admit failure.
- Most remarkable: one epoch produces most of the habits and halves the input tokens; the capability response depends on the starting model.
§1 Introduction (1.0 page)
- Hook: open coding agents are increasingly trained on trajectories of a frontier agent; the recipes report resolve rate, and resolve rate hides the process (motivation only: 2607.06184; recipes: SWE-Gym / SWE-smith / Open-SWE-Traces).
- Gap (reviewer's wording): prior work establishes fingerprintability and distillation-induced similarity (2606.16988, 2604.21255) and studies which trajectories teach well (2606.03461); we add an epoch-dose curve, paired capability effects across starting points, same-task evidence, and a vendor-generation control in one coding harness — plus the curation pipeline that produced the corpus. Then the positioning sentence verbatim.
- One-sentence contribution (above). Approach: curate → train 1/2/3 epochs on two families → evaluate 12 arms → fingerprint everything → anchor → audit.
- Contributions (4 bullets): (1) a documented curation pipeline with an offline rule-impact analysis (D1, D4) and a composite-intervention comparison (D2); (2) dose-response fingerprint analysis with a one-epoch result on the dense line, replicated in direction on the MoE line and within languages (H1, S1–S4); (3) the dissociation of fingerprint transfer from capability across starting points, with the input-token footprint result (H2, H3); (4) a paired detectable-effect analysis for SWE-bench Pro/Verified and a phrasing audit showing that the SUCCESS_CLAIM regex measures wording, not honesty (H4).
- Results preview: the five numbers above. Hero figure: Fig. 1 (two panels). Key citations: 2607.06184, 2606.16988, 2604.21255, 2606.03461, 2504.21798, 2606.16038.
- Front-loading check: title + Fig. 1 + contribution bullets carry the whole story.
§2 Related Work (0.75 page)
- Trajectory SFT for coding agents (SWE-Gym, SWE-smith, Skywork-SWE, SWE-Dev, SWE-Mirror, Kimi-Dev, Devstral, SWE-Lego, Open-SWE-Traces; what makes trajectories effective 2606.03461) — they report scores; we report what is inherited.
- Behavioural fingerprints and distillation-induced similarity (2606.16988, 2604.21255, 2606.17454, idiosyncrasies 2502.12150, subliminal transfer 2507.14805 / 2604.15559 / 2606.11270, distillation detection 2607.09692) — we add dose, capability co-measurement, anchor, vendor comparison.
- Style vs capability in SFT (LIMA, URIAL, 2410.03717, SFT-memorizes 2501.17161, long-CoT distillation) — agentic instantiation.
- Evaluation validity (resolve rate hides process 2607.06184, 2604.02547; flakiness 2605.17281 / 2605.28840; false success 2606.09863; token ≠ cost 2607.12161; contamination 2512.10218 / 2506.12286).
- Organised by question, not paper by paper.
§3 Corpus: generation and curation (1.4 pages incl. floats) — innovation line 1
- Generation: teacher, harness, 32,830 trajectories, hidden-test verification (F2P ∪ P2P), vendor audit.
- Curation funnel figure/table (Fig. 2 / Table 2): the P0 drop rules with thresholds and counts, demote-but-keep rules, message-level loss masks (what each masks and how many rows), same-task cap, length caps; final 8,137 / 425.
- v13 variant: relaxed thinking floor, by-product inclusion,
failed_conclusionmask (regex quoted), 23,238 rows. - Rule-impact table (new, offline;
analysis/data/rule_impact.md): for every rule family — sole and marginal removals, overlap with other rules, the behavioural statistics of removed vs kept trajectories (thinking share, steps, tool mix, Bash heads, write-without-read, truncation pipes), the loss-character mass each mask removes, and the language/repository composition before and after filtering. This is what makes the curation line more than a count table. Delivered 2026-09-02:raw_thinking_lowalone removes 2,450 of 3,435 drops (71 %) without changing the tool mix (JS 0.005) but takes the longest, most think-heavy rows (68 vs 44 steps; think share 0.897 vs 0.838); sourcing rules remove 493 + 45 (+221 pre-pipeline) Bash-heavier rows with poor recovery (leak family 0.24 vs 0.60); demote tier removes nothing; cap −39, inert on concentration (6.66 → 6.69 %); masks 2.77 % of learnable chars (v13 3.43 %, failed_conclusion 0.15–0.94 % over 3,088 rows); composition flat (Go 37 → 38 %, top-15 repos 40.7 → 37.7 %); v13 by-product inclusion is the only composition shift (Go 28 %). All counts reconcile with build_report.json exactly. - What the rules did and did not touch (forward pointers to §5/§6): thinking share (D3, descriptive), harness-error recovery (S2), write-without-read (D4), variant false-success wording (H4), Edit-parameter hallucination is not corpus content.
- Honest note: no single-rule ablation corpus exists; v12 vs v13 is a composite intervention and is presented as such.
§4 Students, evaluation and fingerprint extractor (0.65 page)
- 12 arms table (Table 1 columns: base, corpus, epochs, Pro pass@1, Verified pass@1, cumulative input/task, bash-JS to teacher). Same harness, same instruction, pool as denominator.
- Fingerprint rulers (mix, bigram, Bash heads, thinking position), JS distance, equal-weight trajectory averaging; the split-half floor (same arm, random halves) as the sampling reference in the body, and the temperature-0.6 twins as a temperature-perturbation reference (never called reliability); three gates.
- Two sentences on evaluation resolution (from App. A): with one rollout per task the paired design resolves ≈ 4 pp on Pro and ≈ 5 pp on Verified; differences below that are reported as unresolved.
- Anchor and audit protocols in two sentences.
§5 What the student inherits (1.8 pages incl. floats) — innovation line 2
- 5.1 Dose response (H1): Fig. 1a; one-epoch saturation; layered fingerprint (habit vs strategy) shown on Pro vs Verified; ruler dependence stated in one sentence (TV vs JS).
- 5.2 Anchors (S1): if the held-out 373-task anchor lands, a compact panel in Fig. 3 (same-pool distances, never-trained tasks) and the 88 trained-task anchor in the appendix; if it does not land, the trained-task anchor stays in the appendix, "task-independent" is dropped from every claim, and the memorisation limitation is foregrounded in §8.
- 5.3 The inherited bundle (S2, S3, S4, D3, D4): Fig. 3 multi-row "what moved toward the teacher": Bash head-word shares (
echo,sed,python), recovery@3, post-error thinking pattern, thinking share (corpus reference line) and position deciles, write-without-read, sub-agent share, failure-ending wording. Vendor next-gen row as the descriptive comparison.
§6 What it costs and what it buys (1.15 pages incl. Table 1)
- 6.1 Token footprint (H3): numbers in Table 1 (cumulative input, steps × per-step) with the mechanism in two sentences; Fig. 5 (waterfall + observation sources) in the appendix; output rises; wall-clock flat; token ≠ bill (cite 2607.12161).
- 6.2 pass@1 (H2): Table 1 differences in pp; bare vs post-trained; teleport-excluded column; generation jump; softened interpretation ("consistent with the corpus supplying part of the agentic post-training transition on one bare base; the design does not isolate a general base→instruct mechanism"); D2 composite-intervention row (v13).
- No p-values in the body; one sentence pointing to Appendix A for the paired analysis.
§7 Measurement limits at this evaluation scale (0.3 page)
- H4 in prose: temperature-perturbation flips, the detectable effect at this task count (already introduced in §4), and the audited endings (0/110 admissions; model-annotated by two models unless the human pass lands). Behavioural rulers separate arms that pass@1 cannot; no reliability claim without a same-condition retest.
§8 Conclusion (0.2 page) + Limitations (uncounted) + Ethics/reproducibility statement
- Restatement; limitations: single teacher/corpus/harness, teacher never ran the evaluation pools (anchor; held-out anchor if available), model-annotated endings pending human pass, repo-level exposure (teleport), no single-rule curation ablation, MoE dose curve has two points.
Figure Plan
| ID | Type | Description | Data source | Priority |
|---|---|---|---|---|
| Fig 1 | Hero, two panels | (a) epochs × Bash-JS to teacher (log) with the split-half sampling floor and the vendor next-gen marker; right axis Pro pass@1 — no detectable-effect band (statistics live in the appendix), the two axes visibly separated by marker style; (b) cumulative input per task × Pro pass@1 with three arrows: corpus on post-trained base (left, flat), corpus on bare base (up-left), generation jump (up-right, 2.9× cost); optional narrow curation-funnel ribbon so the figure carries both innovation lines. Caption (reviewer's): "One epoch produces most dense-model behavioral convergence; SFT reduces median input by up to 53 %, while capability response depends on the starting model." | figures/fig1.tex (pgfplots); fp_summary; report_excerpt; page numbers | HIGH |
| Fig 2 / Table 2 | Funnel + rule table | 12,257 → 12,036 → 8,601 → 8,562 → 8,137/425 with rule → threshold → removed; mask kinds with row counts; v13 column | analysis/data/data_pipeline.md | HIGH |
| Table 1 | 12-arm table | base, corpus, epochs, Pro/Verified pass@1, Δ vs native (pp), Pro-minus-teleport Δ, cumulative input, bash-JS | report_excerpt; analysis/stats | HIGH |
| Fig 3 | Multi-row dot plot | "what moved toward the teacher": for each metric, teacher / native / +v12 ×1/2/3 / v13 / vendor next-gen | fp_summary, fp_pooled | HIGH |
| Fig 4 (appendix; compact panel into Fig 3 only if the held-out anchor lands) | Anchor | read-set precision/recall/Jaccard + negative controls (bars), Δsolve vs ΔJaccard scatter | analysis/fp/anchor_diag.json | MEDIUM |
| Fig 5 (appendix) | Waterfall + stacked bars | cumulative input decomposition for three pairs; observation sources (whole-file reads, limited reads, commands ± pipes) | 算_行为分解 outputs; A3B page data | MEDIUM |
| Table 3 (appendix) | Endings audit | strong-form claim rate by arm; two-annotator counts (claim / truncated / admit); RATIONALIZE verification share | fp_summary overclaim; analysis/annot | MEDIUM |
| App. A | Evaluation noise | flips, detectable effects, Holm families, teleport slices, rollout×3 | analysis/stats | HIGH (appendix) |
| App. B | Fingerprint robustness | Hellinger/TV tables, floors, per-language strata, per-arm distributions | analysis/fp; lang_robustness | MEDIUM |
| App. C | Leak audit, protocol errors, regression, compaction, dense vs sparse | existing report pages | LOW |
Citation Plan (all IDs verified in lit/arxiv_verified.md)
- §1: 2607.06184, 2606.16988, 2604.21255, 2606.03461, 2504.21798, 2606.16038, 2509.16941 (SWE-bench Pro).
- §2 trajectory SFT: 2412.21139, 2504.21798, 2506.19290, 2506.07636, 2509.08724, 2509.23045, 2509.25193, 2601.01426, 2606.16038, 2606.03461. Fingerprints/distillation: 2606.16988, 2604.21255, 2606.17454, 2502.12150, 2507.14805, 2604.15559, 2606.11270, 2607.09692. Style vs capability: 2305.11206, 2312.01552, 2410.03717, 2501.17161, 2502.03373, 2501.19393, 2502.03387. Evaluation validity: 2607.06184, 2604.02547 [read before citing], 2605.17281, 2605.28840, 2606.09863, 2607.12161, 2512.10218, 2506.12286, 2506.16170.
- §3: 2601.01426 (error masking as an accuracy lever), 2606.09863 (false success), 2512.10218 (contamination).
- §6: 2607.12161. §7: 2605.17281, 2606.09863.
- SWE-bench Verified [VERIFY citation], Claude Code / Opus 4.8 [VERIFY vendor citation form], Qwen3.5/3.6/3.8 model reports [VERIFY].
Reviewer Feedback (Codex gpt-5.6-sol, xhigh, 2026-09-02; trace traces/paper-plan/)
Scores before fixes: flow 8, alignment 5, missing 6, positioning 7, budget 4, front matter 6, overall 6. Applied: abstract/H2/D2/D3/D4/S2/S3 rewordings; "planned wording" column and provisional D rows; temperature twins renamed a temperature-perturbation reference; §4 carries the detectable-effect sentences; §7 renamed; positioning sentence used verbatim and the brittle "nobody has measured" removed; page budget made float-inclusive (7.25 + buffer) with Fig 4/Fig 5/Table 3 moved to the appendix; title replaced by the reviewer's 10-word proposal; hero figure loses the detectable-effect band; base→instruct interpretation softened; held-out anchor made the claim gate for "task-independent"; new §3 rule-impact table commissioned (analysis/data/rule_impact.md).
Next Steps
- [x] Cross-review this outline (Codex, xhigh) and apply the minimum fixes
- [x] Rule-impact table for §3 (offline; delivered 2026-09-02:
analysis/data/rule_impact.md/.json,tex/tables/tab4_rules.tex) - [x] /paper-figure: Fig 1 (done 2026-09-02:
figures/fig1.pdf, provenance infigures/fig1_data.json) - [x] /paper-figure: Fig 3 (body, done 2026-09-02:
figures/fig3.pdf) - [x] /paper-figure: Fig 4–5 (appendix, done 2026-09-02:
figures/fig4.pdf,figures/fig5.pdf); Fig 2 = funnel table intex/tables/tab2_funnel.tex(drafted) - [x] Table 1 (
tex/tables/make_tab1.py, fully sourced 2026-09-02); Table 2 funnel drafted; Table 3 endings audit skeleton intex/appendix/A_evaluation_noise.tex - [x] /paper-write: draft v1 (2026-09-02) in
tex/— body ends on page 8, appendix A–C with 20 sourced tables, Codex cross-review applied (traces/paper-write/2026-09-02_run01); DATA_NEEDED blanks for val373 / pro100 in §5 and Limitations - [ ] /paper-compile; /paper-claim-audit; /citation-audit
- [x] GPU: held-out anchor (must) — landed 2026-09-06: val373 → 345 oracle-passed tasks × 8 arms (
analysis/anchors/val373/), pro100 grew to 359 Pro tasks run by the teacher (analysis/anchors/pro359/); Appendix E (Tables E1–E5) + body Figure 15; "task-independent" claimed (composite 0.051 → 0.006 at one epoch on never-trained tasks; read precision 0.56 → 0.78 held-out vs 0.50 → 0.80 trained); corrections applied: teacher makes 0 task-management calls on the benchmark (3.6 % on val, 3.2 % corpus), thinking position within 2× floor on the same tasks - [x] Qwen3.8-27B + v12 graft: dropped 2026-09-06 (user decision, "不测了"); the generation comparison stays descriptive with the native Qwen3.8-27B arm only
- [x] Same-condition retest (reviewer item 3): STOPPED 2026-09-06 15:50 at 9/345 by user decision (no further experiments unless strictly necessary; temperature twins + split-half floors stand as the reliability reference) — had started 13:20 on inst1 engine + gpu23 sandboxes: n27 and s256 again on val373 at T=0.8 (
chain_val373.sh LINE=C ARMS="n27 s256" TAG_SUFFIX=-retest); when done, add flip rate and fingerprint test–retest to Appendix A/E and §7 - [ ] GPU: v12 on Qwen3.8-27B (should)