Profile AI execution from trace to verified change¶
Availability: the workflow APIs and examples on this page are included in
0.1.0. Run the checked-in examples from a source checkout:
git clone https://github.com/dipeshbabu/agentloop.git
cd agentloop
uv sync --locked --all-extras --dev
uv run python examples/email_workflow.py --out runs/email-reference
AgentLoop profiles AI-powered software, including agents, classifiers, routing rules and extraction pipelines. Your application chooses its models and executes its decisions. AgentLoop records that execution, connects findings to evidence, and compares an explicit candidate with a baseline under your quality criteria.
Follow one optimization through the evidence¶
- Record the baseline. Use
trace_workflow,WorkflowInfoandStageInfoto identify the workflow, stages and input references. Record physical model calls and generic operations separately, with explicit dependencies where known. Preserve task IDs for later pairing. See workflow tracing. - Define acceptable outputs. Freeze independent labels or a versioned scorer before comparing implementations. Structured quality supports decisions, labels, numbers, extracted fields and matching. Missing outputs remain unavailable; a failed run does not pass by omitting a result.
- Inspect findings. Canonical rules link recommendations to their source spans. Optional semantic investigations can assess repetition, contribution, progress, retries and context relevance. Ordinary analysis reads saved judgments; it never calls a judge automatically.
- Execute an explicit candidate. Change the application and record the same inputs again, or run an offline substitution experiment for one decision step. Declare the baseline, candidates and frozen criteria. Agreement with the baseline is separate from independently measured quality.
- Compare and retain the evidence. Use replay gates and paired studies to assess quality and performance together. Retain failed, unmatched and indeterminate trials. The intervention workflow links original findings and predictions to the change and measured outcome.
A finding is a reason to investigate. Estimated savings are conditional on their recorded assumptions; they are not measured improvements. A passing comparison applies to its supplied cases and criteria. Application owners review the evidence and decide whether to adopt a change.
Choose the evidence contract¶
| Need | Contract and guide |
|---|---|
| Trace an agent or a non-agent workflow | Existing trace_agent, plus generic workflow and stage metadata |
| Assess output correctness | Caller-defined structured quality, including unavailable outcomes |
| Ask an optional semantic question | Typed judgments, with identity, source references, uncertainty and known usage |
| Connect a local model, rule or service | Host-owned adapters and judgment benchmarks |
| Investigate unnecessary semantic work | Evidence-bound semantic findings |
| Compare replacements for one decision | Frozen offline substitution experiments |
| Review before/after evidence | Intervention records, studies and HTML reports |
Native traces retain the existing schema and storage path; ExecutionTrace is
an alias of AgentTrace. Existing agent APIs and default behavior remain
supported. No hosted service or preferred model provider is required. Native
cost and token totals cover recorded model calls, not arbitrary service billing
or all application compute. Missing usage stays unknown. Substitution reports
identify their separate declared decision-step accounting scope.
Judgment execution is explicitly enabled by the caller. The host supplies trusted callbacks or service adapters and controls provider spending and cancellation; timeouts are cooperative. Judgment agreement and confidence are evidence, not independent ground truth. Preserve backend versions and configuration provenance when reviewing a result.
Try the reference workflows¶
Each example uses synthetic inputs, independent fixture labels, local callback implementations, native traces and baseline/candidate evidence. Use a fresh or empty output directory for each command.
| Workflow | Run from the checkout | What to inspect |
|---|---|---|
uv run python examples/email_workflow.py --out runs/email-reference |
Spam, category, priority and routing; required verification retained | |
| Payment review | uv run python examples/payment_review.py --out runs/payment-reference |
False positives, false negatives and explicit manual review |
| Incident triage | uv run python examples/incident_triage.py --out runs/incident-reference |
Severity, inert response proposals and required safety checks |
| Marketplace | uv run python examples/marketplace_workflow.py --out runs/marketplace-reference |
Duplicate, policy, category and final-route quality |
| Batch data | uv run python examples/batch_data_pipeline.py --records 24 --chunk-size 8 --out runs/batch-reference |
Per-record quality, chunk-level pairing and stage distributions |
The examples do not send email, move money, execute infrastructure proposals or call external model APIs. Model-style fees and token counts are fixture units; timings measure local callbacks. Cheap and failing variants expose quality traps. These outcomes verify the contracts, not real-model quality or production savings. The batch guide describes optional prompt capture and report-volume limits.
Apply the workflow to your application¶
Use permission-cleared inputs, source/configuration revisions and independent quality criteria. Set provider budgets before enabling external callbacks. Keep the same task population across conditions and report the complete planned denominator, including failures and missing evidence. Review captured bodies, metadata and references before sharing artifacts.
The generic contracts and references complete roadmap #196. Hardening roadmap #208 covers finding trust, experiment design, volume, onboarding, drift and real-world proof. Real-agent validation in #181 and the runtime roadmap #182 remain separate work; these synthetic examples do not satisfy their empirical requirements.