Skip to content

Context reduction and result reuse experiments

These templates, included in 0.1.0, create ExperimentRunner candidates for the generic experiment contract. Baselines and candidates use the same frozen case inputs, expected answers and pairing keys. Native traces, interventions, budgets, quality gates and JSON/HTML/Markdown exports remain the result format. No production caching or automatic rollout is enabled.

Explicit context changes

context_reduction_runner wraps an existing ExperimentRunner. Configure one to 64 ContextSelection objects, each naming an opaque selection ID, explicit JSON path, source reference and remove or compress action:

from agentloop.context_experiments import ContextSelection, context_reduction_runner

candidate = context_reduction_runner(
    "remove_optional", backend,
    selections=[ContextSelection("optional-notes", ("context", "notes"), "segment:notes:v1")],
    protected_paths=[("context", "required_evidence")],
)

Paths use dictionary keys or nonnegative list indexes. Selection paths cannot overlap each other or protected paths. Array removals use original positions. Inputs are owned copies; unrelated fields and protected context remain unchanged. Missing selections fail by default. Explicit on_missing="keep_original" uses the original request and records that no context change was applied.

Compression requires an explicit trusted synchronous compressor(value, request) and versioned compressor_ref. It returns the existing ExperimentResult, including its usage. Compressor and backend usage must be exclusive: duplicate usage IDs are rejected, known tokens/costs are combined, and unknown components keep the total unknown. Partial compression usage is retained if later work fails. Configure a reservation covering all compression and backend work; the helper does not assume the backend-only bound also covers a compressor.

Optional token counters return ContextTokenCount with an explicit counter_ref. Provider/tokenizer/user-supplied exact counts remain distinct from estimated_words and unavailable counts. Counter references and before/after bases must match for a reduction comparison. A local tokenizer is appropriate; hidden work or fees inside a trusted counter are not automatically intercepted. Include any such work in the host's declared usage/budget accounting. Time checks occur before callbacks, and the generic experiment still rejects late results cooperatively.

Trace metadata agentloop.context_experiment preserves selection IDs/references, path/content hashes, action, replacement hashes, token-count provenance and usage. It does not require raw removed or compressed context. Hashes of predictable sensitive content are not anonymization; use permission-safe opaque references.

context_applied and exact_token_reduction are observations about the transformation, not quality conclusions. Independent frozen quality scoring can reject a shorter prompt. Reports keep estimated token counts labelled as estimated and show quality-preserving subsets alongside the full denominator.

Frozen result reuse

result_reuse_runner uses a bounded, read-only snapshot of prior ReuseEntry results. It does not build a semantic cache, populate a cache on misses, or manage a production cache service. This makes state reproducible across experiment resume.

from agentloop.reuse_experiments import result_reuse_runner

candidate = result_reuse_runner(
    "reuse", backend, entries=frozen_entries,
    key_paths=[("task", "entity_id"), ("task", "locale")],
    key_version="1", data_ref="dataset:v3", invalidation_epoch="epoch-2",
    as_of="2026-09-15T00:00:00Z",
)

The lookup key is the exact canonical hash of the ordered selected values; absent fields differ from explicit nulls. Entries must match key version, backend implementation reference/version, configuration reference, data reference and invalidation epoch. They also require a source reference and valid creation/expiry metadata. Bumping any binding invalidates old entries.

as_of freezes the scenario's evaluation time. It is not a promise that an entry will remain fresh in production. Entries created after that time, expired entries (including the exact expiry boundary), inverted lifetimes and missing provenance are ineligible. Among eligible matches, the newest creation time wins; conflicting outputs at that time are ambiguous and cannot be used. Equal valid duplicate outputs have a deterministic reference tie-break.

Default fallback recomputes through the declared backend. on_unavailable="error" instead fails the candidate. Metadata agentloop.reuse_experiment retains key and entry hashes/references, hit/miss/stale/invalidated/unknown/ambiguous states and whether fallback actually ran. No raw key or cached output is needed in these metadata records. The host retains the private prior-result snapshot and its source artifacts.

A snapshot hit performs no new provider token work. Lookup operating cost remains unknown, and prior cache-fill cost is explicitly excluded from the per-access comparison. Do not treat an empty model profile as zero workflow cost or infer amortized savings without accounting for fill/invalidation/storage work.

Key completeness is a host contract. A declared key can still be too coarse for the task and return a wrong answer. Independent quality checks expose that case; a mechanical hit alone is never counted as quality-preserving reuse.

Shared observations and comparison reports

The generic experiment's bounded agentloop.experiment_observations namespace stores scalar values with observed, estimated or declared kind, units and source references. Entries cannot be rewritten during a trial. Summaries only combine matching kinds, units and source references; mixed or unknown bases stay explicit. Missing attempts remain in every planned denominator.

Boolean observations expose true/false/missing counts, fraction of all planned attempts and an independently quality-preserving true count/fraction. Cache hit rate describes key/provenance matching; quality-preserving hit rate additionally requires the independent task gate. Successful recomputation after a stale/missing entry is task success, not reuse success. Overall replay gates evaluate the complete candidate, including fallback, and do not establish that caching caused a benefit.

Numeric observations show recorded values and conditional quality-preserving summaries. Each candidate report keeps realized native pair deltas separate from the original finding predictions. Recorded-model estimated costs remain separate from exclusive caller-reported runner usage and unknown operating costs.

Offline validation

uv run python -m examples.context_reuse_experiments --out runs/context-reuse
uv run python -m examples.context_reuse_experiments --out runs/context-reuse

The synthetic example compares optional-context removal, harmful required-context removal, valid reuse and an incomplete reuse key. It retains quality regressions even when token counts or callback latency fall. The second invocation executes zero completed slots. The character tokenizer and timings are fixture evidence, not real-model savings. No provider request is made.