CLAIMS_FROM_RESULTS — result-to-claim gate (ARIS)
verdict: PARTIAL reviewer: Codex gpt-5.6-sol, model_reasoning_effort=ultra, thread 01a0603e-7bec-7053-afe1-aeb378eee48c, 2026-09-02 (trace: traces/result-to-claim/2026-09-02_run01/, prompt traces/result-to-claim.prompt.txt, full response traces/result-to-claim.response.md) evidence pre-check: 344/344 cited numbers found in their source files (evidence_precheck.json, built by build_claims.py from the data files themselves) — no hallucinated evidence. integrity audit: not run (EXPERIMENT_AUDIT.json absent) → this verdict is provisional per the skill's rules. inputs judged: NARRATIVE_REPORT.md v1.0 (the ledger as it stood before the judge; corrections listed in §3 were applied afterwards as v1.1).
1. Verdict table
| Claim | Verdict | Conf. | Accepted wording (judge) | Required before submission |
|---|---|---|---|---|
| H1 one-epoch saturation | partial | high | "On SWE-bench Pro, one v12 epoch accounts for most of the observed dense-model convergence in Bash vocabulary and tool-mix/bigram JS; later changes are small, while on Verified only Bash vocabulary and thinking position converge." | held-out same-pool anchor (373 val tasks); difference CIs (not overlapping marginal CIs); alternative distance; intermediate MoE checkpoints for a one-epoch MoE point |
| H2 fingerprint vs capability | partial | high | "Fingerprint proximity changes substantially whether or not pass@1 changes: bare-A3B SFT improves Pro by 8.89 pp but is inconclusive on Verified, while no baseline-relative post-trained comparison is significant and its intervals still permit several-point effects." | Holm over the 111 valid paired tests; MDE re-derivation on 731; teleport-excluded tables; equivalence-style inference instead of "≈0"; v12 on Qwen3.8-27B; drop causal "score comes from the base" wording |
| H3 cost | partial | high | "Under the Claude Code Pro harness, v12/v13 SFT reproducibly reduces cumulative input tokens through shorter runs and more bounded observations, while increasing output tokens; this is a token-use signature, not a general monetary or wall-clock saving." | paired CIs per task; mini-SWE-agent harness check (nice to have); rename to "input-token footprint is part of the fingerprint" |
| H4 measurement | partial | high | "At these sample sizes, one-rollout paired pass@1 does not resolve most observed 0–3 pp SFT differences and task outcomes are temperature-sensitive; behavioural reliability remains unestablished, and SUCCESS_CLAIM measures strong completion phrasing rather than honesty." | final MDE; test–retest reliability of behavioural metrics (note: temperature 0.8 vs 0.6 is not same-condition test–retest — say so); human re-annotation with agreement; do not attribute the v13 −5 pp to the mask (corpus and epochs also differ) |
| S1 same-task anchor | partial | high | "On 88 v12 training tasks, s256 more closely matches teacher read/edit sets and tool mix than n27, while its solve-rate difference is imprecise; these data do not establish that imitation exceeds memorisation." | held-out anchor; reconcile rollout counts (263 vs 296 trajectories; one task has two rollouts); never write "imitation ≫ memorisation" |
| S2 error handling | partial | high | "On Pro, SFT consistently raises three-call protocol-error recovery toward the teacher across both student families; the correlated thinking and rework measures do not yet establish a general error-handling bundle." | task-clustered CIs; report Verified heterogeneity (s128 recovery 0.448 → 0.273); absolute post-error thinking |
| S3 thinking | partial | high | "Across both pools, SFT increases thinking-character share above the pooled teacher reference; thinking-position JS generally moves teacherward, except for the post-trained MoE comparison on Verified." | CIs; held-out anchor; keep the gradient mechanism as a hypothesis only |
| S4 Qwen3.8 comparison | yes | high | "On Pro under this harness, native Qwen3.8-27B matches distilled arms on Bash vocabulary, thinking position, and truncation-pipe use, while remaining different on tool composition, reading, and error recovery; no training-cause inference follows." | none |
| S5 channel narrowing | no | high | "SFT changes channel use in a base-dependent manner: it reduces sub-agent execution for native dense and MoE bases but increases it for the bare MoE base; we make no general channel-narrowing claim." | dropped as a claim; keep one descriptive sentence + the git-history observation (one pool) |
| Overall contribution (§0) | partial | high | "In this single teacher–corpus study, v12 SFT on 8,137 trajectories shifted Pro command vocabulary, protocol-error recovery, and thinking position toward the teacher in two Qwen families and cut median cumulative input tokens by as much as 53 % (with most dense-model fingerprint change after one epoch), whereas pass@1 effects depended on base and pool: +8.9 pp on Pro but an inconclusive +3.4 pp on Verified for one bare base, and −2.2 to +2.8 pp versus native post-trained bases, with no significant exact baseline comparison." | — |
Strongest (judge): Bash/recovery convergence replicated across dense and MoE with large effects; q38 +5.61/+5.8 pp significant on both pools; bare-base +8.89 pp (p = 4.5e-7). Weakest (judge): behavioural reliability asserted not measured; "imitation ≫ memorisation" compares incommensurate, under-powered endpoints; general channel narrowing contradicted by the bare-base direction.
2. Routing (skill Step 4: partial)
- Working claims updated to the accepted wordings above (NARRATIVE_REPORT.md v1.1 §7;
论文骨架.mdrewritten). - Gaps recorded in
findings.md. - Supplementary work, in order: (a) analysis-only fixes — Holm, MDE on 731, bootstrap difference CIs for the four distances, a second distance, test–retest reliability on the two Pro temperature pairs (labelled as temperature sensitivity, not test–retest), teleport-excluded tables, absolute post-error thinking, rollout-count reconciliation; (b) human re-annotation of 240 + 68 endings with two annotators and κ; (c) GPU: held-out 373-task anchor (must), v12 on Qwen3.8-27B (must if it finishes), MoE checkpoints 127/254 (nice), mini-SWE-agent check (nice).
- Re-run this gate once (a)–(c) land; the twelve existing arms are final and are not re-run.
3. Inconsistencies the judge found, and how each was resolved
| Judge's finding | Resolution |
|---|---|
"8.6k trajectories" vs 8,137 in think_overshoot.json | 8,137 is right (train_ms_swift_262144.jsonl has 8,137 rows; 425 val rows). Narrative and骨架 corrected. The 8,601 figure in the 27B report page was an earlier row count. |
| Teacher groups sum to 32,583, not 32,830 | 247 captured trajectories carry no pass/fail verdict in their manifest and belong to no group; scan_teacher_manifests.py re-run to confirm the breakdown (result to be pasted into findings.md). Narrative now states both numbers. |
| Python-reference endpoint 0.0178 is s381, not s256 (0.0195) | Corrected: n27 0.0267 → s256 0.0195 → s381 0.0178. |
| s256 Pro composite is 0.0215, not 0.0213, in the anchor comparison | Corrected. |
| 88-task groups hold 263 (n27) vs 296 (s256) trajectories; one 0.5 solve fraction | Stated explicitly: 2–4 rollouts per task, per-task medians; the 0.5 is a two-rollout task. To be reconciled in the anchor script before the paper. |
| "Only v13–s381 leaves zero" — Pro s256–n27 and s256–s128 Wald CIs also exclude zero; s256–n27 Wald CI conflicts with exact p = 0.054 | Corrected: by exact McNemar p < 0.05 the baseline-relative exceptions are v13 vs s381 (Verified, worse) and s256 vs s128 (Pro, +3.15 pp, p = 0.014); s256 vs n27 Pro is p = 0.054 with a Wald CI just excluding zero. All go through Holm. |
report_excerpt.json lists 132 tests, 21 involving arms with no Verified run | Noted in the narrative: 111 valid tests; the 21 void rows are dropped by build_claims.py's consumer (the paper's test family is 111). |
| "All v12/v13 checkpoints at 1.52–1.80 M" vs 2.1 M for the MoE arm | Corrected to "dense-line checkpoints"; MoE stated separately. |
| 110 regex-negative samples vs "0/109" | The file's tally rows sum to 110 while its summary says 109; 0 admissions either way; the one-count discrepancy is assigned to the human re-annotation. |
| 140 → 92 is −34 %, not −30 % | Corrected. |
4. Consequences for the paper plan
- Framing follows the novelty verdict (see
NOVELTY_REPORT.md): "What agentic SFT buys: rapid behavioural and input-token convergence, while capability gains depend on the starting model." Headline set stays H1–H4 with the accepted wordings; S5 leaves the claim list; S4 is the one claim the judge accepted as written. - "Pass@1 is the wrong endpoint" is a motivation, not a claim (anticipated by arXiv:2607.06184); the measurement claim is the narrower "one-rollout paired pass@1 at 731/500 tasks does not resolve 0–3 pp effects".
- Cost language: "input-token footprint", never "cheaper".