Same-agent harness ablations¶
agentloop study ablation summarizes a frozen host-run experiment. It does not
execute agents, reset tools, score outputs, or turn stored traces into new runs.
The real SQL-agent ablation applies this contract to
counterbalanced local-model executions with independently scored public tasks,
retained early stops and separate tracing/policy overhead measurements. The
synthetic example below remains an offline contract fixture.
Run the offline example¶
uv run python examples/harness_ablation.py --out runs/harness-ablation
uv run agentloop study ablation runs/harness-ablation/bundle.json --out runs/harness-ablation/report.md --json-out runs/harness-ablation/report.json
The example constructs synthetic observations, native traces, an existing study manifest, an immutable intervention record, and JSON/Markdown reports. Its numbers are fixtures, not measured timings, scorer evaluations, provider charges, or evidence that the fixture guard implements a recommended rewrite. The ledger link tests artifact preservation only. For actual execution of local fake calls under controls, use the LangGraph harness example.
There are 24 planned observations and 23 recorded ones. Of four held-out enforce-versus-shadow pairs, one passes quality, one is stopped and fails, one lacks quality, and one is missing. All four stay in the denominator. The faster stopped run appears in descriptive timing but cannot count as quality-preserving improvement. Repeated runs of this fixture produce the same reports.
Freeze the design before collecting outcomes¶
Use the existing application's runner with these four conditions:
| Mode | Configuration | Comparison |
|---|---|---|
uninstrumented |
Original agent, no AgentLoop trace or policy wrapper in the measured path | Reference for trace overhead |
tracing |
Same agent with tracing, controls disabled | Tracing minus uninstrumented |
shadow |
Tracing and one policy evaluated without enforcement | Shadow minus tracing |
enforce |
Same policy/configuration with admission controls enabled | Enforce minus matching shadow; also enforce minus tracing |
Add policies individually before combinations. Each policy set must have a matching shadow/enforce condition and single-policy conditions for every member. Condition identity includes the declared policy version/configuration reference. The report does not choose policies or promote a configuration automatically.
Build a specification with the fields below and freeze it with
AblationProtocol.freeze(specification), imported from
agentloop.ablation_protocol. Save protocol.to_dict() before measurement, then
attach its protocol_hash to each observation. from_dict() verifies that the
saved specification still matches its digest. Returned dictionaries are copies.
Keep a dated external record of the frozen file: its hash and frozen_at field
alone cannot prove preregistration.
| Specification field | Required declaration |
|---|---|
schema_version |
1.0 |
name, workload_id, permission_ref |
Study/workload identity and the owner's permission reference; the loader does not grant or verify permission |
frozen_at, synthetic |
Time with an explicit timezone, and a boolean fixture marker |
versions |
Immutable references for source, model, provider, configuration, scorer, runner, environment, tools, and reset |
tasks |
Task ID to split (pilot or held_out) and input_sha256; both splits are required |
conditions |
Condition name to mode and policies (policy ID to version/configuration reference); empty policies for off/tracing |
schedule |
Slots with task_id, string repetition, cache_condition, and an ordered list of all conditions |
quality_gate |
min_score and max_regression, each in [0, 1] |
bootstrap |
Positive samples up to 10,000, integer seed, and confidence strictly between zero and one |
Freeze the full model configuration, prompts, pricing snapshot, scorer code and scorer configuration through those immutable references. Keep task input and expected-answer artifacts outside the trace if sensitive. Hash the complete task specification, including its constraints; predictable hashes are not anonymization. Identical task digests cannot become separate task IDs. Represent repetitions as separate slots under the same task. A task cannot cross splits. Choose held-out task counts from pilot variability and the predeclared effect size/quality tolerance; small examples remain exploratory.
The pairing key is (task_id, repetition, cache_condition). A slot must contain
each condition once. Counterbalance condition order within split/cache strata;
the report exposes planned position counts and whether each condition's counts
differ by at most one across positions. That descriptive balance check does not
certify absence of carryover or provide an experimental design recommendation.
Declare cache preconditioning/reset procedures and analyze distinct cache states
separately. Provider seeds must be recorded when available; they do not guarantee
deterministic provider behavior.
Record host observations without changing the agent¶
The host runner owns workload permissions, execution, external monotonic timing, state snapshots/restoration, cache preparation, scorer calls, and spend limits. Restore the declared environment before each condition. Measure the same span of work in every condition; score results separately using the frozen scorer. Do not derive the uninstrumented arm's latency from spans or fabricate a trace for it. Record failed, timed-out, cancelled and interrupted attempts as well as successes. Keep planned slots even when no observation survives.
Each observation contains these fields (see the example for a complete record):
- Identity/design:
observation_id,protocol_hash,task_id,repetition,cache_condition,condition, zero-basedposition,started_at, actualversions,reset_confirmed, and optionalprovider_seed(use null if unknown). - Outcome:
status(completed,failed,cancelled,timed_out,stopped,incomplete), optional boundedstop_reason,success(boolean or null), and independentquality_score(number in[0, 1]or null). - Metrics:
latency_ms, totaltokens,cost_usd,model_calls,tool_calls,retries, and aggregatepolicy_eval_ms, insidemetrics. Unknown values are null. Counts are nonnegative integers; durations and costs are finite and nonnegative. Policy evaluation time can sum concurrent work and is not wall-clock overhead; use the matched condition differences to examine overhead. - Provenance:
token_status,cost_status,cost_basis, andtrace_run_id(null when no trace is available). Use the existing token/cost status vocabulary; cost basis isprovider_reported,calculated,mixed, orunknown.
These are host declarations. The reporter cannot verify that the runner used the declared model, reset its environment, applied the intended policy, or ran an independent scorer. Preserve runner logs and harness decision evidence for review. Missing traces do not manufacture decision evidence. Do not put prompts, raw answers, exception text or secrets in IDs, version references or stop labels.
Protocol-hash/version mismatches, observations predating the freeze, unconfirmed resets and wrong order positions remain visible as protocol deviations. They cannot contribute paired metric deltas. Unplanned observations are retained without enlarging planned denominators. Duplicate pairing keys are ambiguous; the reporter never selects the most favorable attempt. Duplicate observation IDs are invalid artifacts.
Interpret quality and measurements¶
The quality gate requires both observations to be completed, explicitly successful, scored above the minimum, and within the declared regression allowance. Decimal score/regression boundaries use the declared values without binary subtraction rounding. A stopped run is conservatively excluded from quality-preserving results even if a scorer says its partial output is acceptable. No-error spans never substitute for task-grounded success. Missing quality is indeterminate.
Each comparison reports accepted, rejected, indeterminate and unmatched counts
against all planned pairs. quality_acceptance_rate is the observed accepted
fraction of that denominator; missing pairs are not asserted to be failed tasks.
Raw/descriptive metrics retain failed and stopped attempts. Separate
quality-preserving metrics use only gate-accepted pairs while retaining their
missing counts and full planned denominator. Neither view authorizes a universal
policy-benefit or quality guarantee.
Tokens contribute only with exact/empty provenance and an available count. Provider-billed costs and calculated costs have separate summaries and intervals; they are never pooled. Provider-reported complete charges do not require exact token counts. Calculated costs require exact/empty token provenance as well as complete pricing. Partial, unknown and mixed-basis charges remain in the raw observations but cannot become zero or an evaluable paired cost. Cost savings do not compensate for rejected or missing quality.
Deltas are candidate minus baseline. Reports separate pilot and held-out tasks
and cache strata. For each metric they show paired summaries and equally weighted
task means: available repetitions are averaged inside each task, then tasks are
weighted equally. A seeded percentile bootstrap resamples those task means, not
individual seeds. Fewer than two evaluable tasks produces insufficient_tasks.
Intervals describe the observed task sample under task exchangeability; they do
not fix selection/missing-data bias, scorer error, multiplicity, small-sample
coverage or workload representativeness. Metric-specific task/pair denominators
are retained in JSON and Markdown, including the quality-eligible subset.
Link existing evidence¶
A bundle contains schema_version: "1.0", the frozen protocol, the complete
observations list, optional study_manifest path, and interventions paths.
Paths resolve relative to the bundle. An empty observations list is valid and
reports every scheduled comparison as missing.
The CLI rejects resolved report paths that collide with each other or any bundle,
study, trace or ledger file it read, guarding against choosing an input as output.
The linked study uses only instrumented conditions and pairing keys
["task_id", "repetition", "cache_condition"]. Observation trace references must
match the study's condition and pairing metadata. Existing intervention records
are loaded unchanged; their source IDs and trace fingerprints must match that
study. Duplicate ledger artifacts are rejected. Unlinked study traces are listed
explicitly. Synthetic markers from linked traces/records propagate to the report
even if the protocol forgot its marker.
The embedded study remains descriptive. Its older pair-level bootstrap, if requested in that manifest, is not the task-level ablation interval. Keep original prediction snapshots before outcomes; do not regenerate them after observing a candidate. Publishing a real case study also requires the permission-cleared bundle, measured onboarding/instrumentation friction, retained regressions, and the owner-reviewed task/scorer/spend protocol from #181.