Skip to content

Empirical calibration with attribution and failures retained

The September 2026 calibration artifact contains 184 original finding registrations. It preserves the earlier agent-routing outcomes without assigning one combined intervention's effect to each of its findings. A separate single-decision experiment supplies attributable latency observations, but none of its candidate outcomes preserves quality. No runtime coefficient or policy is changed, and the fitted diagnostic factor is not a deployment recommendation.

Inventory and attribution

Source Registrations Treatment
Revised #181 pilot and held-out routing study 168 98 selected findings from 42 combined interventions; 70 rejected findings retained without claimed outcomes
New single-decision experiment 16 One original routing finding per intervention; four fitting pairs and twelve held-out pairs

All source traces, original predictions and immutable ledger records are retained in the checksummed evidence bundle. The native report has nine separate estimator/workload/configuration cohorts. No synthetic contract fixtures enter empirical fitting.

The 42 earlier interventions replaced all model steps and contain two or three original findings each. The existing calibrator marks their 98 selected registrations combined_intervention_unattributed and duplicate_outcome. These labels mean the same combined outcome cannot be credited repeatedly; they do not mean 98 independent executions occurred. The records are neither split nor rewritten to force attribution. Other findings remain rejected with their selection reasons and original predictions visible.

The retired first onboarding protocol remains available in #181's bundle and is outside these revised-workflow cohorts. The #188 count-cap study has no matching recommendation-based estimator attribution and is not fitted here. Inventory completeness applies to the explicitly named cohorts, not every possible workload.

Attributable experiment

The additional workload is one model decision that returns a numeric answer to a public GSM8K question. It is not another autonomous-agent benchmark. Eight questions are chosen by a fixed hash order, excluding indices used in #181: two fitting tasks and six held-out tasks, each repeated twice under both models. Exact bounded numeric normalization supplies independent quality labels.

The baseline and candidate use the same pinned model artifacts as #181, with temperature zero, disabled prompt caching and a 32-token output cap. This short-answer constraint is part of the tested configuration, not a claim about either model's general mathematical ability. Each trace has one model call and one selected route_to_smaller_model prediction. The original finding is saved with a post-write timestamp journal before any paired candidate runs. Outcomes receive a separate post-write journal after the native ledger is saved.

All 32 attempts and 16 interventions are retained. The baseline answers 6/16 correctly; the candidate answers 0/16 correctly. Six candidate responses are malformed/truncated under the output cap; other unsuccessful answers remain completed-but-incorrect outcomes. The native independent quality gate rejects all 16 pairs. Failed outputs are not removed from raw latency calibration.

Raw latency diagnostics

The original estimator is route_to_smaller_model version 1.0, with the saved heuristic sum_duration_ms * latency_fraction and latency fraction 0.25. The native diagnostic fit averages repetitions within each fitting task, then fits a multiplicative factor through the origin. Its factor is 1.258062, based on only two fitting tasks. That identifiability floor is not adequate evidence for general calibration.

Split Independent tasks / pairs Mean original prediction Mean realized raw savings Mean original signed error Mean scaled signed error
Fit 2 / 4 289.920 ms 381.827 ms -91.907 ms -17.090 ms
Held out 6 / 12 301.081 ms 253.683 ms +47.398 ms +125.096 ms

Signed error is prediction minus realized savings; realized savings is baseline minus candidate latency. The factor shifts held-out mean bias farther positive. The original held-out signed-error interval is approximately -294.242 to 414.101 ms. Intervals resample task means, not repetitions as independent tasks. Small samples, fixed condition order, shared hardware and public-benchmark training overlap limit interpretation.

These are raw resource diagnostics, including wrong and malformed answers. The quality-preserving subset has zero eligible observations. No cost factor is fitted: self-hosted operating cost is unavailable for every run. Zero paid-provider spend is recorded separately and is not substituted for measured operating cost. The original predictions, coefficients and ledger contents remain unchanged.

Time provenance and freshness

For the earlier study, original archive-file last-write times were captured from the retained execution directory and checked to precede candidate starts. Their prediction hashes match the already-published evidence. These filesystem times are disclosed provenance, not independent proof of preregistration. If only a relocated archive is available, the inventory builder leaves unrecoverable times unknown rather than backdating them.

The new cohort uses explicit post-write journals. The artifact's declared as_of is 2026-09-24T01:34:40.318364+00:00, with valid_until 2026-10-01T01:34:40.318364+00:00. This historical window does not refresh when the report is read and never authorizes activation. Any model, provider, workload, scorer, policy, environment or quality-gate change requires renewed evidence even within that window.

Reproduce without model calls

python examples/real_agent_study/archive.py research/empirical-calibration-2026-09/index.json --out /tmp/agentloop-calibration-evidence
agentloop study calibrate /tmp/agentloop-calibration-evidence/calibration/manifest.json --out /tmp/calibration-report.md --json-out /tmp/calibration-report.json

Use the recorded AgentLoop source/version. The native calibrator reads saved predictions and outcomes; it does not rerun today's estimator, reprice historical calls, execute models or modify policies. Relocation changes source-path fields and the artifact hash, but numerical results and cohort decisions are reproducible. Use the frozen manifest for reproduction rather than inventing archive timestamps from newly extracted files.

examples/real_calibration_study.py prepares and explicitly executes new local single-decision trials. examples/empirical_calibration.py builds the historical inventory and delegates all diagnostics to the existing native calibrator. The model and data sources retain the licenses and attribution from #181. GSM8K's MIT license is included. No model weights or inference binaries are redistributed, and required CI makes no model calls.