Trace AI workflows and decision pipelines¶
These APIs are included in 0.1.0. Start with the AI execution profiling guide for source setup and the path from tracing through quality checks, findings and paired comparisons.
The email pipeline reference executes spam, category, priority and routing stages with paired quality, replay and semantic evidence. The payment-review reference demonstrates structured review labels, explicit abstention and failure retention without financial actions. The incident-triage reference evaluates inert response and rollback/escalation proposals without changing infrastructure. The marketplace reference preserves per-stage and final-route quality evidence for duplicate, policy and category decisions. The batch data reference profiles many records in generic workflow chunks, with explicit extraction/matching quality and retention tradeoffs.
Use structured quality contracts to compare the caller's routing, extraction and matching outputs through replay and paired studies.
trace_workflow profiles classifier/rule/retrieval/transform pipelines using the
same lifecycle, native trace schema, stores and HTTP routes as trace_agent.
ExecutionTrace is an alias of AgentTrace. Existing agent APIs and default
agent reports remain supported.
AgentLoop records the application's work and caller-declared outcomes. The host application owns classification, routing and execution; this API adds no graph runner, queue, scheduler or automatic policy enforcement.
Try the offline pipeline¶
uv run python examples/workflow_trace.py --out runs/workflow-trace
uv run python examples/workflow_trace.py --linear --out runs/workflow-trace-linear
The example profiles synthetic functions and writes native JSON, analysis JSON, HTML and OTLP. It verifies metadata round trips without retaining the input message. These are contract fixtures, not application usefulness or performance evidence. A branching dependency declaration describes dataflow; the example's application code still controls execution order.
Declare workflow and stage identities¶
from agentloop import (
StageInfo,
WorkflowInfo,
set_workflow_outcome,
trace_operation,
trace_workflow,
)
with trace_workflow(
"Message routing",
workflow=WorkflowInfo(
"message-router",
"git:reviewed-revision",
input_schema_ref="schema:message:1",
output_schema_ref="schema:route:1",
input_ref="task:message-123",
),
task_id="message-123",
example_id="routing-case-123",
) as trace:
with trace_operation(
"Classify",
kind="classifier",
stage=StageInfo("classify", "classifier-config-v3"),
depends_on=[],
) as classification:
# category = application_classifier(message)
category = "billing" # Placeholder for the application's result.
classification.set_outcome(category, output_ref="result:category-123")
with trace_operation(
"Route",
kind="rule",
stage=StageInfo("route", "routing-rules-v2"),
depends_on=[classification.event_id],
) as routing:
# application_route(category)
routing.set_outcome("routed", output_ref="result:route-123")
set_workflow_outcome(trace, "routed", output_ref="result:route-123")
WorkflowInfo and StageInfo are immutable declarations. Both carry identity,
version and optional input_schema_ref, output_schema_ref, input_ref and
output_ref. A stage can declare a logical kind separately from the physical
operation. References are opaque strings: recording one does not retrieve it,
validate its schema or prove that an artifact exists. Hashes can be references;
predictable hashes are not anonymization.
Workflow task/example IDs also live in ordinary trace metadata for existing study pairing. Explicit arguments accept strings or integers and reject conflicting metadata. These APIs require and capture no raw input/output bodies. Metadata is caller-provided JSON; use references and safe labels for shareable artifacts.
Operations and resource scope¶
The existing taxonomy now also recognizes rule, classifier, transform and
external_service, alongside model, tool, retriever, memory, reranker,
guardrail, evaluator, workflow, agent and retry.
| Kind | Native event category | Accounting |
|---|---|---|
model |
model_call |
Existing model/token/cost metrics |
tool |
tool_call |
Existing tool-call metrics |
retry |
retry |
Existing retry metrics |
| Other kinds | operation |
Operation counts and timing, without invented tool/model calls |
For a physical model call used for classification, use kind="model" and
StageInfo(kind="classifier", ...). Both token counts are required for exact
provenance. Missing usage stays unavailable; non-model token usage is rejected
before a context body runs.
For dynamic SDK usage, keep existing model instrumentation and attach
operation_metadata("model", stage=StageInfo(...)), or call record_operation
after receiving normalized usage. Generic contexts do not inspect SDK responses.
Keep grouping spans distinct from physical model calls to avoid double counting.
Cost and token totals retain their recorded model-call scope; arbitrary
service billing and non-model computation are not included in those totals.
Callbacks, status and outcomes¶
record_operation accepts completed timing, native ok/error status, optional
model usage, stage/reference metadata, dependencies and an explicit captured
trace/parent_id. An explicit trace does not inherit an unrelated ambient parent.
Records use the same AgentEvent validation and storage path as other traces.
operation_metadata and workflow_metadata support integrations that already own
their tracing lifecycle, without framework imports. workflow_metadata accepts
an explicit execution status for completed host observations.
Contexts restore trace/parent bindings and propagate original errors and
cancellation. Error class names are recorded; raw exception details require
capture_error_detail=True. OperationSpan.set_outcome accepts a bounded label
and optional output reference; its saved result cannot be rewritten after close.
Workflow status is running, then completed, failed, cancelled or
interrupted; metadata-only integrations may use unknown. Completion means the
traced scope returned, not that task quality passed. Keep worker lifetimes inside
the owning trace, entering/exiting contexts in the same execution context. For
streams, use completed-span records with captured context or an existing adapter
that restores context on each resume.
Dependencies and compatibility¶
Dependencies reference span/event IDs, not stage names. Omission is undeclared;
[] explicitly declares a root. Parent spans describe containment. Dependencies
describe prerequisites and do not establish read/write safety or permission to
parallelize.
When explicit stage dependencies are present, graphs use parent and declared
edges instead of inventing causal edges from timestamps. Declaration coverage,
invalid references, unresolved IDs and cycles remain visible. Invalid/unresolved
declared graphs have no critical-path duration; direct critical_path() raises
an explicit error. Validation reflects graph mutations. Without declarations,
the existing inferred-sequence behavior remains identified as such. Legacy agent
graphs retain their behavior.
Ordinary workflow repetition does not activate agent-specific runaway-loop or tool-oscillation heuristics. Other findings keep their assumptions and gain stage context in evidence; a finding remains a hypothesis requiring validation.
No native schema or storage migration is needed. Versioned agentloop.workflow
and agentloop.stage metadata carry the declarations. Unknown kinds remain intact
and analyze as unknown; unknown metadata versions remain in native artifacts
and appear as unsupported in projections.
Graphs, analysis/findings, replay, studies, native JSON, existing stores/HTTP routes, HTML/Markdown and dashboard views preserve the metadata. Studies cannot infer success for known failed/cancelled/interrupted workflows merely from absent error spans; unfinished/unknown completion is indeterminate. Completed execution without task evidence still does not establish task quality.
OTLP carries workflows on resource metadata and stages on spans. Generic operations use native AgentLoop model/usage attributes rather than claiming GenAI operations. Physical legacy model/tool/retry exports keep their conventions. Original event IDs preserve dependency references through repeated round trips.