计数基线结论与风险口径

Counts-only baseline: independent result-to-claim review

claim_supported: yes, strictly for the requested descriptive development-point claim. Confidence: high for that bounded claim and numerical reconstruction. The result is SUPPORTED_DESCRIPTIVE_DEV_BASELINE_POINT_ONLY. It is not a demonstrated safe/generalizing policy, a new method result, or a reason to advance the paper/test gate.

This is a reused-context, same-family, provisional review. The same reviewing thread previously reviewed the separate repaired narrative experiment. This task was judged from the counts-only baseline's original seed reports, saved predictions/models and dev/ext dataset. It is not a fresh zero-context or cross-family audit. The prior narrative NO neither determines nor contradicts this verdict: these are different scores and opportunity sets.

Intended claim and actual scope

The bounded claim supplied for review is that an expanded-data counts-only existing baseline has a favorable point on the generator-token frontier in the full development replay. That statement is supported. Any stronger statement that harm is at most 1% in the population, quality is noninferior on new tasks, or the baseline improves measured net compute remains unestablished.

The completed suite comprises seeds 0/1/2, four score families, two policy menus, and two calibration constraints: 48 stored tables, representing 16 distinct numerical tables. Two extension-trained histogram gradient-boosted classifiers predict current correctness and native terminal correctness. gain = p_now − p_final and failure_termination = 1 − p_final are derived scores, not extra fitted heads. Every score submits the current patch at its first available threshold crossing; otherwise it follows the native endpoint. Failure termination includes no restart and is not a full Fail-Fast/Restart-Smart or EET reproduction.

The inputs are fixed public counts/status features plus repository encoding; there is no self-probe or text-model feature in this counts-only run. The 300-task / 600-attempt extension plan has 599 captured attempts. Training uses the existing eligible complete subset: 580 attempts across 296 tasks, including four native-only attempts with zero decision-loss mass; 13,178 decision rows have supervised loss. The stored weight sum 0.9932432432432432 retains that zero-loss mass and is not an accidental normalization defect. Existing unknown/incomplete extension outcomes were excluded according to the recorded eligibility policy, not filled with negative labels. This is not evidence of full extension-label coverage.

Development includes all 40 tasks / 160 attempts and 4,091 decision rows. All development labels needed here are known; the native-only attempt stays in the denominator. Five hash-based task folds calibrate thresholds or at-most-two-component episode mixtures using only the other four folds. The risk choices are empirical H ≤ .01 and the supplemental net-loss constraint H−R ≤ .01; net loss does not bound harm separately. Thresholds come from the fixed recorded menu plus native. This evaluator does not perform nested selection among the four score families and two menus.

Independent checks

verify_replay.py imports no production evaluator. It directly reconstructs first crossings from the authoritative dev/ext dataset and saved score arrays, joins the selected known current/native outcomes, and reads cumulative actor costs at the selected row. It checked 40,480 trajectory/threshold candidates (485,760 metric cells), the base-fold component choices, and all point metrics. It validated task assignments and paired native costs.

I read each original approximately 21 MB seed results.json, not merely the root's projected evidence-check summaries. The reports contain all 2,000 bootstrap sample metrics. All 672 point values and 1,344 interval endpoints across 48 tables agree with reconstruction or direct quantile calculation. The reports' 1,344,000 stored bootstrap scalar values were read. For a bounded independent calibration check, I regenerated the exact task draws and recalibrated the fixed first 64 draws plus the original point for all 16 distinct combinations: 1,040 evaluations and 5,200 fold calibrations, including exhaustive feasible mixture pairs. Those reproduce the saved numbers. I did not independently regenerate all 2,000 calibrations; all their saved quantiles and cross-seed equality were checked.

The separate bounded saved-model check under model-check/ passes 261/261 assertions without a learned-model refit. It verifies all 27 public count/repository features, extension-only indices/weights and model configuration. All six heads' manual frozen-tree and saved predict_proba outputs reproduce probabilities exactly on 17,692 rows each. All 1,200 trees' extension-training leaf memberships and weighted leaf Newton updates match; the maximum update discrepancy is 1.39e-17. The three seeds' four prediction arrays and tree arrays are elementwise identical, and their complete numerical tables, fold choices and bootstrap samples are identical. These are three completed configured runs of effectively deterministic training, not three independent experimental replications. They do not triple the development sample size or establish robustness to training randomness.

The 288-value deterministic existence check used unchanged per-table JSON projections because the original large-file textual scan was stopped for pathological runtime. I inspected that adaptation and used original raw reports for my verification. The projection is an indexing convenience, neither replacement data nor semantic evidence by itself.

Favorable deterministic point and its limits

Deterministic failure_termination chooses threshold 0.9 in all five folds, under both calibration constraints. It stops 45/160 attempts and retains 77/160 successes, exactly the native count. It has one harmed attempt and one rescued attempt: H = R = 0.625%, hence ΔQ = 0. Harm remains visible despite being balanced by rescue.

Generator input falls from 3,229,760.34375 to 2,655,043.21875 tokens per attempt, a 17.7944% reduction. Output falls from 22,257.8125 to 19,484.8375, a 12.4584% reduction. Mean controller decision count is 19.6625, or 3,146 decision evaluations in this development replay. Decision count is not CPU time.

The harmful first crossing is sphinx-doc__sphinx-10673-0, fold 3, step 26, dataset row 3501: score 0.9020469383841186, selected outcome 0 versus native 1, actor costs 269,273 input / 2,608 output versus native 3,597,665 / 16,547. The rescued first crossing is sympy__sympy-15875-2, fold 2, step 74, row 3919: score 0.9204152626515802, selected outcome 1 versus native 0, actor costs 2,473,941 / 32,257 versus native 3,629,797 / 37,655. Selected full-tree and patch hashes and every one of the 160 first-stop records are saved in failure_termination_first_stops.json. No test labels were opened for this attribution.

Under harm calibration, the conditional 95% input-saving interval is [8.4046%, 31.2074%] and output-saving interval [5.2100%, 22.5553%]. However, H's interval is [0%, 3.125%], and paired ΔQ's interval is [-2.5, +1.25] percentage points. Thus positive conditional token savings are supported; a population harm ceiling of 1% and quality noninferiority are not. Under net-loss calibration, the same original point has different recalibration uncertainty: H [0%, 5%], ΔQ [-5, +0.625] points, and input savings [9.1094%, 33.8350%]. Identical original thresholds do not make the two bootstrap procedures interchangeable.

These are paired whole-task bootstrap intervals with threshold/mixture recalibration. They condition on the fitted extension models and recorded trajectories. They exclude extension-model training uncertainty, new actor-generation uncertainty and the extensive prior development/project adaptation. Their nominal confidence level is not a post-selection or population safety certificate.

Complete menu, without selecting a favorable row for inference

All values below are the common seed0/1/2 values. Percentages are absolute rates except token savings; ΔQ is in percentage points versus native Q = 48.125%. Mixture rows are analytical expectations over one threshold sampled before the episode.

The deterministic current-correctness score also meets the 1% empirical harm limit but loses 0.625 quality points. Deterministic terminal-correctness and gain scores exceed the harm and 1% net-loss limits on held-out folds. Every mixture row exceeds 1% harm and 1% net quality loss at its held-out point. Their conditional 95% harm intervals are entirely above 1%, and their ΔQ intervals are entirely negative. The larger failure-termination mixture input savings, 20.0285% under harm calibration and 21.3399% under net-loss calibration, therefore do not constitute quality-preserving improvements. Fitting an empirical constraint on training folds does not guarantee it on their held-out tasks.

Naming deterministic failure termination as the most favorable observed point among this completed menu is an exploratory comparison. The score families were recorded before their fits, but deciding which family is best from their aggregate out-of-fold development performance is still full-development family selection. No nested family selector or independent final comparison is supplied here.

What is supported, missing, and next

Supported: a useful completed baseline measurement on repaired development data, including a real favorable point for deterministic failure termination and accurately retained unfavorable alternatives. Missing for broader claims: independent tasks after a full freeze, training/generation uncertainty where relevant, selection-aware comparison, a true population-risk assessment, and actual controller/runtime costs. The present measurements are generator input/output tokens reported separately; no controller CPU, net GPU, wall-time saving or price saving was measured. Training/evaluation wall time in the run logs does not provide online controller cost.

This baseline does not include P3's additional native-quality, output and CPU-envelope constraints. It also uses a different opportunity set. It therefore is not a matched P3 comparison, and neither a superiority claim for P3 nor a negative judgment about the running P3/0.6B jobs follows. They are outside this review. A later common comparison must make the opportunity/action, costs and selection constraints explicit before interpretation; do not retrofit a winning family to outcomes already seen and call it independent evaluation.

[INTEGRITY: WARN] The inherited EXPERIMENT_AUDIT.json concerns the older repaired paired-stop P2 and its named dataset/results. It is not a new comprehensive audit of this counts-baseline suite. This review adds bounded model/arithmetic verification, not a blanket experiment-integrity PASS. The dataset hash is recorded in this review's receipts; no raw labels or outputs were changed.

Suggested claim: “On the repaired 40-task, 160-attempt development cohort, the existing counts-based failure-termination baseline saved 17.79% of generator input tokens and 12.46% of output tokens while preserving the observed solve rate, with one harmful and one rescuing stop. This exploratory point does not establish a 1% population harm guarantee; conditional task-bootstrap harm intervals reach 3.125% under harm-based calibration.”

Route: retain this suite as a verified development baseline and a requirement for future fair comparisons. Do not inflate it into a new method contribution, independent-seed replication, population safety, or matched P3 result. No extra experiment is needed to verify this bounded descriptive point. Further experiments are needed only for those broader claims, under a separately frozen comparison and independent evaluation plan. Keep test sealed and continue separately authorized method work; paper readiness and the overall project goal remain unproved.

The reviewer authored this conclusion. No production source, raw data/result, active job, GPU service, or test artifact was changed. Verification used the existing CPU environment only. Full receipts are independent_replay_receipt.json, failure_termination_first_stops.json, and model-check/receipt.json in this trace, with reproducible helper scripts adjacent.