NOVELTY_REPORT — novelty-check verdict (ARIS)
verdict: PROCEED WITH CAUTION — score 7/10 — closest prior work arXiv:2604.21255 (ACL 2026 main) — no ABANDON paper named reviewer: Codex gpt-5.6-sol, model_reasoning_effort=xhigh, thread 01a06047-cf46-7f93-bfb5-05fdecbc69e9, 2026-09-02 (trace traces/novelty-check/2026-09-02_run01/; dossier NOVELTY_DOSSIER.md) literature record: 42 arXiv IDs, every one verified through the arXiv API on 2026-09-02 (lit/arxiv_verified.md, raw lit/arxiv_verify.jsonl, script lit/verify_arxiv.sh); initial scan by a background literature agent (36 papers), extended by 6 that surfaced during ID verification.
What the caution requires (judge's list, adopted)
- Complete the held-out same-pool anchor (373 never-trained tasks with teacher trajectories).
- Add confidence intervals for the fingerprint distances and a second distance measure.
- Quantify behavioural test–retest reliability (the Pro temperature pairs are temperature sensitivity, not same-condition retest — label them so).
- Apply Holm across the 111 paired tests.
- Make the bare-vs-post-trained qualification central to the framing.
Positioning adopted
Framing (b): What agentic SFT buys: rapid behavioural and input-token convergence, while capability gains depend on the starting model.
One-sentence positioning (judge's wording): *Prior work shows that agent behaviors can be fingerprinted and made more teacher-like; we show where that convergence saturates, what operational habits and input-token footprint it transfers, and why it can be sharply dissociated from paired solve-rate gains on post-trained coding models.*
Anticipation constraints to honour in Related Work: 2606.16988 (coding-agent fingerprints, JSD, distilled pair) anticipates H1/S4; 2604.21255 (controlled distillation similarity) anticipates H1 and the behavioural half of H2; 2606.03461 conceptually anticipates H2; 2607.06184 anticipates "resolve rate hides process", so "pass@1 is the wrong endpoint" is motivation, not novelty. Still to read closely: 2604.02547 *Beyond Resolution Rates* (verified to exist; overlap not assessed).
Full reviewer report (verbatim)
Novelty Check Report
Proposed Method
This is an empirical measurement study, not a new distillation method. It combines controlled agentic SFT across epoch doses, dense and MoE students, bare and post-trained starting points, paired SWE-bench evaluation, trajectory fingerprints, same-task replay, and input-token decomposition.
The central finding is novel as a conjunction, but not in its broadest wording. Controlled distillation-induced behavioral similarity is already shown by 2604.21255, while coding-agent fingerprinting and student–teacher JSD are already shown by 2606.16988. Neither paper appears to combine:
- a 1/2/3-epoch saturation curve;
- behavioral convergence measured alongside paired capability effects;
- the contrast between bare and post-trained bases;
- a same-task training anchor; and
- a vendor-generation control in the same harness.
The closest overall prior is 2604.21255; the closest domain-and-measurement prior is 2606.16988.
Core Claims with Closest Paper and Delta
| Claim | Closest paper | Reviewer-verifiable delta and required wording |
|---|---|---|
| H1: one-epoch fingerprint saturation | 2604.21255; 2606.16988 | Prior work already shows distillation increases behavioral similarity and that coding agents have measurable fingerprints. Claim novelty in the controlled epoch-dose saturation, replication across two architectures and language strata, and separation between command/thinking habits and task-dependent tool structure—not in discovering behavioral transfer. |
| H2: fingerprint convergence dissociates from capability | 2604.21255; 2606.03461 | The new comparison is that nearly identical fingerprint collapse accompanies +8.89 pp on a bare base but effects below the paired detection floor on post-trained bases, while a vendor-generation jump improves score without matching all teacher dimensions. Say capability gains are conditional on the starting model, not that SFT never transfers capability. |
| H3: input-token footprint transfers | 2607.12161 | The delta is a training-linked, replicated mechanism: fewer whole-file reads and more truncation reduce cumulative input while checkpoint score rankings disagree. Replace "cost is a fingerprint" with "input-token footprint is part of the fingerprint"; 2607.12161 already establishes that token reduction need not reduce billed cost. |
| H4: measurement limits | 2605.17281; 2606.09863; 2607.06184 | The delta is a paired SWE-bench MDE analysis plus same-weight task flips and an intervention showing wording changes without honest failure admission. Restrict the claim to "pass@1 is insensitive at this sample size"; behavioral test–retest reliability remains to be measured. |
| S1: imitation exceeds demonstrated memorisation | 2501.17161; 2512.10218 | Direct replay of 88 literally trained tasks shows strong, significant process imitation but only an imprecise +6 pp solve estimate. Do not claim absence of memorisation or "imitation ≫ memorisation": the design cannot exclude an approximately 15 pp effect. |
| S2: error-handling reflexes transfer | 2604.15559; 2602.17037 | Unlike a single injected habit or recovery tool, the study measures naturally occurring protocol recovery, parameter changes, and edit rework converging across student lines. Present these as dimensions of the fingerprint, since generic behavior transfer is already anticipated. |
| S3: thinking quantity overshoots while position converges | 2502.03373; 2501.19393; 2502.03387 | The quantity-versus-position decomposition in agent trajectories appears new. Keep the corpus-gradient explanation explicitly hypothetical because there is no ablation. |
| S4: vendor post-training matches selected dimensions | 2606.16988; 2606.17454 | Comparative agent fingerprinting is anticipated; the delta is the dimension-by-dimension comparison against controlled community distillation and capability/cost in the same harness. With one opaque vendor model, this must remain descriptive and non-causal. |
| S5: channel narrowing | 2604.16027 | The potentially new observation is discrete action-channel change tied to corpus presence rather than semantic output homogenization. Evidence currently rests on one genuinely absent channel; describe "narrowing" rather than a general pruning law. |
The four most important anticipation constraints are:
- 2606.16988 anticipates H1 and S4, including coding-agent fingerprints, attribution, JSD, and a distilled student–teacher comparison.
- 2604.21255 anticipates H1 and the behavioral half of H2, because controlled LoRA distillation already increases tool-use similarity.
- 2606.03461 conceptually anticipates H2 by separating trajectory characteristics and teaching efficacy, although it asks which traces teach rather than what students inherit.
- 2607.06184 anticipates the motivation behind H2/H4 that resolve rate hides process; therefore "pass@1 is the wrong endpoint" cannot itself be the novelty claim.
Closest Prior Work
| Paper | Overlap | Remaining delta |
|---|---|---|
| 2604.21255, When Agents Look the Same (ACL 2026 main) | Controlled distillation-induced tool-use similarity | Coding domain, dose saturation, capability dissociation, anchor, vendor control |
| 2606.16988, Agent trajectories as programs | Coding-agent fingerprints and distilled student–teacher JSD | Controlled SFT intervention, dose curve, paired capability and same-task evidence |
| 2607.06184, What Resolve Rate Hides | Process diagnostics beyond resolve rate | Teacher-relative training intervention and behavioral dose response |
| 2606.03461, What Makes Interaction Trajectories Effective | Trajectory structure and teaching efficacy | Measures inherited habits rather than selecting effective traces |
| 2606.17454, Dissecting model behavior through agent trajectories | Large-scale cross-vendor SWE behavior analysis | No teacher reference or SFT intervention |
| 2607.12161, Token Reduction Is Not Cost Reduction | Direct challenge to token-as-cost framing | Training-induced input-token mechanism and behavioral linkage |
| 2606.09863, From Confident Closing to Silent Failure | False-success language | Intervention plus manual audit separating wording from honesty |
| 2605.17281, ContractBench | Agent flakiness | Paired SWE-bench power and detection-floor analysis |
No paper in the supplied record contains the full headline result. In particular, the published closest neighbor, 2604.21255, lacks the coding setting, capability co-measurement, epoch dose curve, same-task anchor, and vendor-generation comparison. There is therefore no named ABANDON paper.
I cannot responsibly add a missing neighbor beyond the permitted verified record. The unresolved paper most worth close reading is already listed: 2604.02547, Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure; its venue and exact overlap are [UNVERIFIED] here.
Overall Novelty Assessment
Score: 7/10 Recommendation: PROCEED WITH CAUTION
Key differentiator: The paper jointly demonstrates rapid behavioral and input-token convergence, conditional capability effects across starting points, and a vendor control using paired runs in one coding-agent harness.
Main risk: Reviewers can correctly say that both behavioral fingerprinting and distillation-induced similarity are already known. The remaining contribution depends on the dissociation being credible, but task-independent teacher convergence currently lacks the planned held-out same-pool anchor, behavioral reliability is not yet quantified, and the uncorrected test family weakens inference.
The caution would be substantially resolved by completing the 373-task held-out anchor, adding fingerprint confidence intervals and a second distance measure, quantifying test–retest reliability, applying Holm correction, and making the bare-versus-post-trained qualification central.
Suggested Positioning
Use framing (b), revised to:
What agentic SFT buys: rapid behavioral and input-token convergence, while capability gains depend on the starting model.
This is safer than (a), whose generic territory is occupied by 2604.21255 and 2606.16988, and than (c), which overgeneralizes the measurement conclusion and overlaps 2607.06184. A defensible one-sentence positioning is:
Prior work shows that agent behaviors can be fingerprinted and made more teacher-like; we show where that convergence saturates, what operational habits and input-token footprint it transfers, and why it can be sharply dissociated from paired solve-rate gains on post-trained coding models.
NOVELTY: score=7/10, verdict=PROCEED WITH CAUTION, closest=2604.21255, abandon_paper=none
Addendum 2026-09-03 — second literature round (ARIS research-lit, sources: web)
Twenty-three further papers were verified (arXiv API + verify_papers.py, verdict PASS) and folded into Related Work; the full table with each paper's findings and our delta is lit/related_scan_2026-09-03.md. The two nearest statements of our thesis found in this round are Gu et al. 2026 (arXiv:2602.01611, trajectory-SFT scores conflate tool semantics with a memorised interface; epoch-wise the score rises while robustness falls) and Ma et al. 2026 (arXiv:2605.21996, "the student inherits … any flaw in its intermediate steps", with shorter and less redundant rollouts after curated training). Neither measures teacher-relative fingerprints with paired pass@1 on issue resolution, and neither has an epoch dose, so the 2026-09-02 verdict stands: PROCEED WITH CAUTION, closest 2604.21255, no ABANDON paper. Reviewer-requested references (Gudibande 2023, AgentTuning, FireAct, Agent-FLAN, Miller 2024, SWE-RL) are now cited.