Guided optimization¶
The agentloop improve commands connect task capture, baseline findings, a
developer-applied change, and a quality-checked before/after report. These commands
are included in AgentLoop 0.2.0 and newer.
Install or upgrade with python -m pip install --upgrade agentloop-profiler.
Your agent runs in its own environment through an explicit command supplied to
capture. The command receives one JSON request on stdin per task attempt and
writes a native AgentLoop trace to the requested path.
Start with a runnable project¶
agentloop improve init --out agentloop-demo
cd agentloop-demo
python agentloop_runner.py --check
agentloop improve capture --out baseline -- python agentloop_runner.py
agentloop improve start --baseline baseline/capture.json --out optimization
Initialization writes a data-only agentloop.toml, two independent task examples,
a Python runner, a local tool example, and AGENTLOOP.md. It makes no network
requests and refuses to replace existing files. The example performs repeated
local CPU work on fictional notes, with all traces explicitly marked synthetic.
Inspect optimization/plan.md. In agentloop_example.py, replace
return {"notes": [lookup_note(key) for _ in range(3)]} with:
agentloop improve capture --out candidate -- python agentloop_runner.py
agentloop improve finish --session optimization/session.json --candidate candidate/capture.json --change "Reuse one lookup per request" --out results
Open results/report.html. CPU timing varies; acceptance is an observed outcome.
For a rejection example, return {"notes": ["wrong"] * 3}, capture into a fresh
wrong directory and finish into rejected. The correctness check must reject it
regardless of its speed. Keep both result directories.
Connect an existing Python agent¶
From your existing project, run:
The callable receives one JSON input and returns finite JSON. Async callables are supported. For another signature, edit the generated runner. Install the agent and AgentLoop into the runner's Python environment. Keep its existing model/tool instrumentation, or add the SDK decorators. An outer agent span alone does not expose internal repeated calls.
Replace agentloop_tasks.json with your task inputs and independently chosen
expected answers. Do this before baseline capture. The runner never receives the
expected answers. --tasks FILE continues to work and overrides the project
config; --project FILE selects another config. Config files select task data.
Every capture still requires an explicit runner command after --.
The generated adapter exports a finalized failed trace when the agent raises.
Capture reports each failed task and its diagnostic. Use
agentloop improve capture --show-runner-output --out debug-capture -- python agentloop_runner.py --debug
for a traceback; this may reveal application data. An import failure suggests
checking the environment and runner import. A missing trace suggests exporting
the finalized trace to request["trace_path"]. A timeout suggests inspecting the
agent before increasing --timeout.
Define tasks and correctness first¶
Save a workload such as this in tasks.json. Choose expected values independently
of your agent's answers.
{
"schema_version": "1.0",
"name": "Repository metadata questions",
"repetitions": 2,
"tasks": [
{
"id": "package-name",
"input": {"question": "What is the distribution name?"},
"expected": {"answer": "agentloop-profiler"},
"scorer": {"type": "json_subset"}
}
]
}
Supported scorers are exact_match, contains, json_subset, required_fields,
and bounded glob. Each planned baseline and candidate answer must pass its
check. Expected answers and scorer configuration stay out of the runner's stdin
request. The supported bound is 1,000 planned attempts with 1–20 repetitions.
Adapt your agent¶
An adapter around your application can follow this pattern. Replace the
my_agent import with your implementation and retain your existing instrumented
model and tool calls.
import json
import sys
import agentloop
from my_agent import answer
request = json.load(sys.stdin)
with agentloop.trace_agent("repository_agent") as trace:
trace.metadata["output"] = answer(request["input"])
trace.export_json(request["trace_path"])
The request contains task_id, zero-based repetition, input, and an absolute
trace_path. The trace must have finalized timing. Save the final observed output
in metadata.output so the quality scorer can evaluate it. Existing native model
outputs are also supported through the usual quality-report lookup.
Capture and review the baseline¶
agentloop improve capture --tasks tasks.json --out runs/baseline -- python your_agent.py
agentloop improve start --tasks tasks.json --baseline runs/baseline/capture.json --out runs/optimization
Read runs/optimization/plan.md. It shows the selected finding family, affected
spans, observed evidence, suggested changes, assumptions, and validation criteria.
It includes the baseline quality result and the native finding rankings by
evidence, quality risk and estimated benefit. Estimates retain their provenance
and missing inputs. They remain hypotheses until candidate measurements exist.
The default focus is the first baseline finding family. Select another observed
family with --focus cache_context, for example.
From AgentLoop 0.2.3, start also compares repetitions of each task. The
repetition_detour finding marks an attempt that took more model steps or tool
calls than the median of its other repetitions and used at least 25% more input
tokens. Runtime replaces tokens only when traces record no token counts. Its
evidence lists the extra tool calls and the intermediate model steps, each of
which usually resends the whole context. Confidence is medium when two or more
attempts in the task set take such a detour. Select it with
--focus repetition_detour. Single-run rules cannot see this pattern, because
each detour looks normal on its own. Use at least two repetitions per task.
start saves original diagnoses, selected finding IDs, trace snapshots, task
definitions, and gate settings in a content-hashed session.json before any
candidate capture. The default objective is latency. Use --objective cost to
require a measured cost improvement, or --minimum-improvement 10 to require a
10% improvement in the selected objective.
Apply a change and compare the same tasks¶
Edit your agent, then run it against the unchanged task set.
agentloop improve capture --tasks tasks.json --out runs/candidate -- python your_agent.py
agentloop improve finish --session runs/optimization/session.json --candidate runs/candidate/capture.json --change "Describe the change applied to the agent" --out runs/results
Open runs/results/report.html. The report shows per-attempt quality, timing,
token provenance, model/tool counts, available model-cost estimates, and native
replay gate decisions. Builds after 0.2.3 show paired timing medians in the CLI,
HTML, and Markdown.
Trace latency covers the instrumented execution and drives latency gates. Runner
wall time covers process launch through exit, including startup, trace export,
and cleanup after interruption. It can help explain a small end-to-end gain when
the traced change is much faster. Runner wall time is descriptive.
Timing medians use completed, correct pairs with available measurements on both sides. Each metric lists how many pairs it could measure out of the full planned set. Failed attempts retain any recorded timings in their individual rows. New captures use a monotonic clock. Older captures with valid start and end timestamps show the derived duration and its clock source. Missing timings show as unavailable.
JSON and Markdown accompany the HTML. Pairs with matching frozen findings also produce native intervention records. Available baseline and candidate traces produce a paired study. Study views retain the original trace fingerprints.
Acceptance requires every planned pair to have completed execution, passing baseline and candidate quality checks, complete analysis, passing native replay gates, and a positive measured improvement in the selected objective. Unknown cost stays unavailable. A latency comparison can pass with an indeterminate, non-gating cost estimate. A required cost improvement fails when pricing or usable usage is missing.
Judge noisy agents across all pairs¶
The default --decision per-pair rejects a change when any single pair is slower
or costlier. Agents that call hosted models vary from run to run, so one noisy
pair can reject a change that helps on almost every attempt. For those agents,
freeze an aggregate decision when starting the session:
agentloop improve start --tasks tasks.json --baseline runs/baseline/capture.json --decision aggregate --out runs/optimization
Aggregate decisions require AgentLoop 0.2.3 or newer. The decision is part of the hashed session, so it cannot be changed after candidate results are known.
Every pair must still complete, pass its quality checks on both sides, have complete analysis, and pass the replay gates that are not about speed or cost, such as retry and quality gates. The speed and cost gates are judged across all pairs instead:
- the median paired improvement in the selected objective must be positive and
at least
--minimum-improvement; - more pairs must improve than regress;
- the other metric's median regression must stay within its allowed regression when every pair can measure it.
The report lists each pair's measurements and failed performance gates for review. It also gives a one-sided sign-test p-value, which is reported for context and does not gate the decision. Use enough repetitions for the median to mean something, and combine aggregate decisions with an interleaved control.
Compare against an interleaved control¶
Interleaved controls require AgentLoop 0.2.2 or newer.
A plain candidate capture runs after the baseline, so cache warm-up, host load or
provider drift can look like an improvement. When you can keep the unchanged
agent runnable, for example in a separate git worktree, capture it as a control
alongside the candidate:
git worktree add ../agent-before HEAD
agentloop improve capture --tasks tasks.json --out runs/paired --control-arg python --control-arg ../agent-before/your_agent.py -- python your_agent.py
agentloop improve finish --session runs/optimization/session.json --candidate runs/paired/capture.json --change "Describe the change applied to the agent" --out runs/results
Repeat --control-arg once for each argument of the control runner. Each
planned attempt runs the control and the candidate back to back. The control
runs first on even attempt indexes and second on odd ones. Use an even number
of attempts, through repetitions, so each condition runs first equally often.
The control capture is stored in runs/paired/control/ and its manifest hash is
recorded in the candidate capture.
finish then measures the candidate against the control instead of the frozen
baseline. Findings, the focus and the gates stay frozen from the original
baseline. The control must pass the same quality checks. Intervention records
link the control's matching findings. The report states which comparison basis
it used, and each case records whether the control ran first.
Run an existing correctness check¶
Builds after 0.2.3 can require a trusted local Python verifier alongside the task's built-in scorer. Use the built-in scorer for basic output shape and the verifier for application behavior, such as checking a code change against your tests or checking that a retrieval answer cites the expected document.
Save this simple citation check as verify_citation.py. It assumes the task
input names the expected source and the trace output includes its citation.
import json
import sys
request = json.load(sys.stdin)
output = request["output"]
passed = isinstance(output, dict) and output.get("source") == request["input"]["source"]
raise SystemExit(0 if passed else 1)
agentloop improve start --tasks tasks.json --baseline runs/baseline/capture.json --out runs/optimization --verifier verify_citation.py
agentloop improve finish --session runs/optimization/session.json --candidate runs/candidate/capture.json --change "Reuse one lookup per request" --out runs/results --verifier verify_citation.py
Capture the candidate after start, as usual. The verifier receives one JSON
object on stdin with schema_version, task_id, zero-based repetition,
input, and the recorded output. It can call your own test code. Exit zero
means pass. Nonzero exit, timeout, cancellation, or unavailable output fails
verification. Output from the verifier process is discarded. Reports show the
status and the exit code or timeout reason.
start checks baseline outputs and freezes the entry script's SHA-256 and
--verifier-timeout (30 seconds by default). finish requires explicit
selection of matching source and rechecks the comparison baseline or control
and the candidate. The task manifest and session metadata contain only data.
The verifier runs with your account's permissions and working directory using
the AgentLoop Python environment. Pin the test fixtures and dependencies
separately. Their contents are outside the entry-script hash. Verification
runs after capture, outside trace and runner timing.
Failures, evidence and execution¶
capture retains every planned attempt, including failed, timed-out, cancelled,
missing-trace and unstarted attempts. finish retains these cases in its
denominator. It returns exit code 1 when the change fails its checks and saves
the report for inspection. Interrupted or incomplete Hermes turns fail the
capture-integrity checks.
Each command writes to a fresh directory. Completed captures and frozen sessions are kept for review. Trace hashes, task/input hashes, distinct run identities, and candidate capture times are checked before comparison. Runner fingerprints include the explicit argv and file-based script content hashes. Retain your source revision, model configuration, and environment alongside these artifacts for reproduction. Hashes establish consistency of supplied evidence.
Execution uses the explicit argv after -- with shell expansion disabled.
Each attempt has a timeout, with an owned process group/tree terminated on timeout
or interruption. The runner inherits the caller's environment and performs its
ordinary application actions. JSON workload files contain task data and built-in
scorer configuration. Standard output and error are discarded during capture by
default. Pass --show-runner-output to stream them when diagnosing a failing runner.
Raw task inputs, expected answers, traces, and outputs are retained in the local artifacts. Review them before sharing. HTML escapes supplied text and shows metrics and decisions. A fixed-order comparison describes the supplied tasks and run conditions. An interleaved control reduces order effects but does not remove other differences between conditions. Repeat across independent tasks and inspect variability before generalizing a result.
Reproduce the Hermes example with local inference¶
The checked-in runner answers three questions about this repository's Python metadata. The task set declares two repetitions and independent JSON-subset checks. The baseline repairs a free-text response into the required JSON contract through a second model turn. The candidate requests constrained JSON from its first turn and uses the same quality checks. Its report is retained whether the change passes or is rejected. A second candidate also checks the returned value against the authoritative local TOML data, repairing mismatches through local parsing. This is application-side validation of structured metadata queries. The external expected answers are held fixed across conditions.
A final condition deliberately asks for an incorrect answer and omits the local value check to exercise the external quality gate.
On Linux or WSL, prepare the pinned Hermes v0.22.7 environment and export the observer as described in the runtime pilot. Then run the following from the AgentLoop checkout.
python scripts/prepare_hermes_model.py --out runs/hermes-local-model
python scripts/run_hermes_adoption.py --hermes-root /path/to/pinned/hermes --plugin-dir /path/to/exported/plugin --model-runtime runs/hermes-local-model/runtime.json --out runs/hermes-adoption
The preparation downloads checksum-verified llama.cpp b10964 and the official
Qwen2.5-0.5B-Instruct Q4_K_M model from pinned URLs. Inference runs on CPU through
an owned loopback server. Profiles and provider settings are isolated per attempt.
The runner uses real Hermes dispatch and normalized model usage. The application
example explicitly retains public repository context and model responses for
inspection.
protocol.json records artifact hashes, decoding settings, request counts, warm-up,
and fixed run order. results/ retains the JSON-only proposal, validated-results/
contains the source-checked proposal, and rejected/ contains the incorrect-answer
control. Model responses and any application-side repair remain visible in the
native traces. Results measure structured output, source-value validation, and
avoidance of the model format-repair call. Self-hosted operating cost remains
unknown. This small controlled workload supplies a reproducible product example.
The task timer covers context loading, model turns, value checks and format repair after Hermes initialization.
The Hermes CI job executes the same guided commands with actual local inference, checks the incorrect-answer rejection, and uploads the complete reports and evidence. The original deterministic runtime pilot continues to cover streaming, tools, retries, interruption, privacy, and delegation.