Skip to content

Guided optimization

The agentloop improve commands connect task capture, baseline findings, a developer-applied change, and a quality-checked before/after report. These commands are included in AgentLoop 0.2.0 and newer.

Install or upgrade with python -m pip install --upgrade agentloop-profiler. Your agent runs in its own environment through an explicit command supplied to capture. The command receives one JSON request on stdin per task attempt and writes a native AgentLoop trace to the requested path.

Start with a runnable project

agentloop improve init --out agentloop-demo
cd agentloop-demo
python agentloop_runner.py --check
agentloop improve capture --out baseline -- python agentloop_runner.py
agentloop improve start --baseline baseline/capture.json --out optimization

Initialization writes a data-only agentloop.toml, two independent task examples, a Python runner, a local tool example, and AGENTLOOP.md. It makes no network requests and refuses to replace existing files. The example performs repeated local CPU work on fictional notes, with all traces explicitly marked synthetic.

Inspect optimization/plan.md. In agentloop_example.py, replace return {"notes": [lookup_note(key) for _ in range(3)]} with:

value = lookup_note(key)
return {"notes": [value] * 3}
agentloop improve capture --out candidate -- python agentloop_runner.py
agentloop improve finish --session optimization/session.json --candidate candidate/capture.json --change "Reuse one lookup per request" --out results

Open results/report.html. CPU timing varies; acceptance is an observed outcome. For a rejection example, return {"notes": ["wrong"] * 3}, capture into a fresh wrong directory and finish into rejected. The correctness check must reject it regardless of its speed. Keep both result directories.

Connect an existing Python agent

From your existing project, run:

agentloop improve init --agent my_agent:answer
python agentloop_runner.py --check

The callable receives one JSON input and returns finite JSON. Async callables are supported. For another signature, edit the generated runner. Install the agent and AgentLoop into the runner's Python environment. Keep its existing model/tool instrumentation, or add the SDK decorators. An outer agent span alone does not expose internal repeated calls.

Replace agentloop_tasks.json with your task inputs and independently chosen expected answers. Do this before baseline capture. The runner never receives the expected answers. --tasks FILE continues to work and overrides the project config; --project FILE selects another config. Config files select task data. Every capture still requires an explicit runner command after --.

The generated adapter exports a finalized failed trace when the agent raises. Capture reports each failed task and its diagnostic. Use agentloop improve capture --show-runner-output --out debug-capture -- python agentloop_runner.py --debug for a traceback; this may reveal application data. An import failure suggests checking the environment and runner import. A missing trace suggests exporting the finalized trace to request["trace_path"]. A timeout suggests inspecting the agent before increasing --timeout.

Define tasks and correctness first

Save a workload such as this in tasks.json. Choose expected values independently of your agent's answers.

{
  "schema_version": "1.0",
  "name": "Repository metadata questions",
  "repetitions": 2,
  "tasks": [
    {
      "id": "package-name",
      "input": {"question": "What is the distribution name?"},
      "expected": {"answer": "agentloop-profiler"},
      "scorer": {"type": "json_subset"}
    }
  ]
}

Supported scorers are exact_match, contains, json_subset, required_fields, and bounded glob. Each planned baseline and candidate answer must pass its check. Expected answers and scorer configuration stay out of the runner's stdin request. The supported bound is 1,000 planned attempts with 1–20 repetitions.

Adapt your agent

An adapter around your application can follow this pattern. Replace the my_agent import with your implementation and retain your existing instrumented model and tool calls.

import json
import sys

import agentloop
from my_agent import answer

request = json.load(sys.stdin)
with agentloop.trace_agent("repository_agent") as trace:
    trace.metadata["output"] = answer(request["input"])
trace.export_json(request["trace_path"])

The request contains task_id, zero-based repetition, input, and an absolute trace_path. The trace must have finalized timing. Save the final observed output in metadata.output so the quality scorer can evaluate it. Existing native model outputs are also supported through the usual quality-report lookup.

Capture and review the baseline

agentloop improve capture --tasks tasks.json --out runs/baseline -- python your_agent.py
agentloop improve start --tasks tasks.json --baseline runs/baseline/capture.json --out runs/optimization

Read runs/optimization/plan.md. It shows the selected finding family, affected spans, observed evidence, suggested changes, assumptions, and validation criteria. It includes the baseline quality result and the native finding rankings by evidence, quality risk and estimated benefit. Estimates retain their provenance and missing inputs. They remain hypotheses until candidate measurements exist. The default focus is the first baseline finding family. Select another observed family with --focus cache_context, for example.

From AgentLoop 0.2.3, start also compares repetitions of each task. The repetition_detour finding marks an attempt that took more model steps or tool calls than the median of its other repetitions and used at least 25% more input tokens. Runtime replaces tokens only when traces record no token counts. Its evidence lists the extra tool calls and the intermediate model steps, each of which usually resends the whole context. Confidence is medium when two or more attempts in the task set take such a detour. Select it with --focus repetition_detour. Single-run rules cannot see this pattern, because each detour looks normal on its own. Use at least two repetitions per task.

start saves original diagnoses, selected finding IDs, trace snapshots, task definitions, and gate settings in a content-hashed session.json before any candidate capture. The default objective is latency. Use --objective cost to require a measured cost improvement, or --minimum-improvement 10 to require a 10% improvement in the selected objective.

Apply a change and compare the same tasks

Edit your agent, then run it against the unchanged task set.

agentloop improve capture --tasks tasks.json --out runs/candidate -- python your_agent.py
agentloop improve finish --session runs/optimization/session.json --candidate runs/candidate/capture.json --change "Describe the change applied to the agent" --out runs/results

Open runs/results/report.html. The report shows per-attempt quality, timing, token provenance, model/tool counts, available model-cost estimates, and native replay gate decisions. Builds after 0.2.3 show paired timing medians in the CLI, HTML, and Markdown. Trace latency covers the instrumented execution and drives latency gates. Runner wall time covers process launch through exit, including startup, trace export, and cleanup after interruption. It can help explain a small end-to-end gain when the traced change is much faster. Runner wall time is descriptive.

Timing medians use completed, correct pairs with available measurements on both sides. Each metric lists how many pairs it could measure out of the full planned set. Failed attempts retain any recorded timings in their individual rows. New captures use a monotonic clock. Older captures with valid start and end timestamps show the derived duration and its clock source. Missing timings show as unavailable.

JSON and Markdown accompany the HTML. Pairs with matching frozen findings also produce native intervention records. Available baseline and candidate traces produce a paired study. Study views retain the original trace fingerprints.

Acceptance requires every planned pair to have completed execution, passing baseline and candidate quality checks, complete analysis, passing native replay gates, and a positive measured improvement in the selected objective. Unknown cost stays unavailable. A latency comparison can pass with an indeterminate, non-gating cost estimate. A required cost improvement fails when pricing or usable usage is missing.

Judge noisy agents across all pairs

The default --decision per-pair rejects a change when any single pair is slower or costlier. Agents that call hosted models vary from run to run, so one noisy pair can reject a change that helps on almost every attempt. For those agents, freeze an aggregate decision when starting the session:

agentloop improve start --tasks tasks.json --baseline runs/baseline/capture.json --decision aggregate --out runs/optimization

Aggregate decisions require AgentLoop 0.2.3 or newer. The decision is part of the hashed session, so it cannot be changed after candidate results are known.

Every pair must still complete, pass its quality checks on both sides, have complete analysis, and pass the replay gates that are not about speed or cost, such as retry and quality gates. The speed and cost gates are judged across all pairs instead:

  • the median paired improvement in the selected objective must be positive and at least --minimum-improvement;
  • more pairs must improve than regress;
  • the other metric's median regression must stay within its allowed regression when every pair can measure it.

The report lists each pair's measurements and failed performance gates for review. It also gives a one-sided sign-test p-value, which is reported for context and does not gate the decision. Use enough repetitions for the median to mean something, and combine aggregate decisions with an interleaved control.

Compare against an interleaved control

Interleaved controls require AgentLoop 0.2.2 or newer.

A plain candidate capture runs after the baseline, so cache warm-up, host load or provider drift can look like an improvement. When you can keep the unchanged agent runnable, for example in a separate git worktree, capture it as a control alongside the candidate:

git worktree add ../agent-before HEAD
agentloop improve capture --tasks tasks.json --out runs/paired --control-arg python --control-arg ../agent-before/your_agent.py -- python your_agent.py
agentloop improve finish --session runs/optimization/session.json --candidate runs/paired/capture.json --change "Describe the change applied to the agent" --out runs/results

Repeat --control-arg once for each argument of the control runner. Each planned attempt runs the control and the candidate back to back. The control runs first on even attempt indexes and second on odd ones. Use an even number of attempts, through repetitions, so each condition runs first equally often. The control capture is stored in runs/paired/control/ and its manifest hash is recorded in the candidate capture.

finish then measures the candidate against the control instead of the frozen baseline. Findings, the focus and the gates stay frozen from the original baseline. The control must pass the same quality checks. Intervention records link the control's matching findings. The report states which comparison basis it used, and each case records whether the control ran first.

Run an existing correctness check

Builds after 0.2.3 can require a trusted local Python verifier alongside the task's built-in scorer. Use the built-in scorer for basic output shape and the verifier for application behavior, such as checking a code change against your tests or checking that a retrieval answer cites the expected document.

Save this simple citation check as verify_citation.py. It assumes the task input names the expected source and the trace output includes its citation.

import json
import sys

request = json.load(sys.stdin)
output = request["output"]
passed = isinstance(output, dict) and output.get("source") == request["input"]["source"]
raise SystemExit(0 if passed else 1)
agentloop improve start --tasks tasks.json --baseline runs/baseline/capture.json --out runs/optimization --verifier verify_citation.py
agentloop improve finish --session runs/optimization/session.json --candidate runs/candidate/capture.json --change "Reuse one lookup per request" --out runs/results --verifier verify_citation.py

Capture the candidate after start, as usual. The verifier receives one JSON object on stdin with schema_version, task_id, zero-based repetition, input, and the recorded output. It can call your own test code. Exit zero means pass. Nonzero exit, timeout, cancellation, or unavailable output fails verification. Output from the verifier process is discarded. Reports show the status and the exit code or timeout reason.

start checks baseline outputs and freezes the entry script's SHA-256 and --verifier-timeout (30 seconds by default). finish requires explicit selection of matching source and rechecks the comparison baseline or control and the candidate. The task manifest and session metadata contain only data. The verifier runs with your account's permissions and working directory using the AgentLoop Python environment. Pin the test fixtures and dependencies separately. Their contents are outside the entry-script hash. Verification runs after capture, outside trace and runner timing.

Failures, evidence and execution

capture retains every planned attempt, including failed, timed-out, cancelled, missing-trace and unstarted attempts. finish retains these cases in its denominator. It returns exit code 1 when the change fails its checks and saves the report for inspection. Interrupted or incomplete Hermes turns fail the capture-integrity checks.

Each command writes to a fresh directory. Completed captures and frozen sessions are kept for review. Trace hashes, task/input hashes, distinct run identities, and candidate capture times are checked before comparison. Runner fingerprints include the explicit argv and file-based script content hashes. Retain your source revision, model configuration, and environment alongside these artifacts for reproduction. Hashes establish consistency of supplied evidence.

Execution uses the explicit argv after -- with shell expansion disabled. Each attempt has a timeout, with an owned process group/tree terminated on timeout or interruption. The runner inherits the caller's environment and performs its ordinary application actions. JSON workload files contain task data and built-in scorer configuration. Standard output and error are discarded during capture by default. Pass --show-runner-output to stream them when diagnosing a failing runner.

Raw task inputs, expected answers, traces, and outputs are retained in the local artifacts. Review them before sharing. HTML escapes supplied text and shows metrics and decisions. A fixed-order comparison describes the supplied tasks and run conditions. An interleaved control reduces order effects but does not remove other differences between conditions. Repeat across independent tasks and inspect variability before generalizing a result.

Reproduce the Hermes example with local inference

The checked-in runner answers three questions about this repository's Python metadata. The task set declares two repetitions and independent JSON-subset checks. The baseline repairs a free-text response into the required JSON contract through a second model turn. The candidate requests constrained JSON from its first turn and uses the same quality checks. Its report is retained whether the change passes or is rejected. A second candidate also checks the returned value against the authoritative local TOML data, repairing mismatches through local parsing. This is application-side validation of structured metadata queries. The external expected answers are held fixed across conditions.

A final condition deliberately asks for an incorrect answer and omits the local value check to exercise the external quality gate.

On Linux or WSL, prepare the pinned Hermes v0.22.7 environment and export the observer as described in the runtime pilot. Then run the following from the AgentLoop checkout.

python scripts/prepare_hermes_model.py --out runs/hermes-local-model
python scripts/run_hermes_adoption.py --hermes-root /path/to/pinned/hermes --plugin-dir /path/to/exported/plugin --model-runtime runs/hermes-local-model/runtime.json --out runs/hermes-adoption

The preparation downloads checksum-verified llama.cpp b10964 and the official Qwen2.5-0.5B-Instruct Q4_K_M model from pinned URLs. Inference runs on CPU through an owned loopback server. Profiles and provider settings are isolated per attempt. The runner uses real Hermes dispatch and normalized model usage. The application example explicitly retains public repository context and model responses for inspection.

protocol.json records artifact hashes, decoding settings, request counts, warm-up, and fixed run order. results/ retains the JSON-only proposal, validated-results/ contains the source-checked proposal, and rejected/ contains the incorrect-answer control. Model responses and any application-side repair remain visible in the native traces. Results measure structured output, source-value validation, and avoidance of the model format-repair call. Self-hosted operating cost remains unknown. This small controlled workload supplies a reproducible product example.

The task timer covers context loading, model turns, value checks and format repair after Hermes initialization.

The Hermes CI job executes the same guided commands with actual local inference, checks the incorrect-answer rejection, and uploads the complete reports and evidence. The original deterministic runtime pilot continues to cover streaming, tools, retries, interruption, privacy, and delegation.