Skip to content

Harness decision evidence

An enabled harness records control decisions independently of model/tool spans. When a trace is active at admission, its metadata["agentloop.harness"] contains the decisions and historical policy declarations. A stream's completion stays with that captured trace even if another trace is active when it closes. Disabled wrappers add no evidence and remain the original callable.

Run the offline example:

uv run python examples/harness_evidence.py --out runs/harness-evidence
agentloop analyze runs/harness-evidence/shadow.json --html runs/harness-evidence/shadow-cli.html

The example writes traces, detached evidence JSON, and HTML for all three modes. Disabled and shadow execute the synthetic tool once; enforce denies it before dispatch. This demonstrates capture and control flow, not an optimization benefit.

Envelope 1.1

The namespace has four fields:

Field Contract
schema_version Independent harness-evidence version, 1.1; readers also accept 1.0
policies Immutable-at-capture declarations indexed by policy configuration hash
decisions Records indexed by their hdec_ identity
capture_errors Safe diagnostics when delivery to a trace failed

run.export_evidence() returns an independent copy, including decisions captured without an active trace. validate_evidence(artifact) validates a detached artifact; read_evidence(trace) additionally checks ownership of its trace references. JSON object order is not execution order: use hook_sequence within a harness run and policy_order within a hook. HTML uses those fields explicitly.

Normal trace-context propagation rules still apply. Bind context explicitly with bind_trace_context when dispatching in a new thread. Calls made without an active trace remain in the detached run artifact with unavailable trace/span references; they are not silently attributed to a different trace.

Policy declarations retain ID, version, configuration hash, priority, hooks, and actions. They are captured when the harness is created. Later configurations do not overwrite earlier snapshots. A shared policy snapshot is stored once per hash, rather than once per decision. Identical duplicate writes are idempotent; different content at an existing decision identity or policy hash raises an error.

Decision fields

Fields Meaning
decision_id, hook_id Stable decision identity and shared hook/effect identity
run_id, call_id, branch_id Explicit harness run, invocation, and branch
trace_id, span_id Captured trace and active parent span at admission; null when unavailable
policy_id, policy_version, policy_config_hash, harness_config_hash Historical policy/configuration provenance
boundary, phase, mode Model/tool/iteration/completion, before/after, and shadow/enforce
requested_action, resolved_action This proposal and the resolution across matching policies
outcome, reason_code, origin Proposal disposition, bounded reason, and policy/intrinsic-harness origin
execution_status, dispatched Observed execution status and whether invocation was attempted
evidence_refs Caller-declared opaque identifiers; no automatic prompt or argument forwarding
conflicting_decision_ids, retry_of Conflicting proposals in the same hook and an explicitly declared prior retry decision
hook_sequence, policy_order Recorded ordering, independent of JSON dictionary order
timing UTC start, policy evaluation duration, and shared hook evaluation duration
budget_snapshot A typed budget snapshot, or null when unavailable; never an invented zero budget
evaluation_status unverified; a decision alone is not an outcome comparison
feedback Optional explicit diagnostic text, limited to 256 characters in schema 1.1

Decision identity is SHA-256 of canonical JSON containing identity version 1.0, run ID, invocation ID, boundary, phase, and policy ID. Identity excludes outcome and timing so an attempted overwrite with changed evidence is a conflict. Invocation IDs distinguish actual repeated calls; retry links do not reuse identities. The shared hook identity omits policy ID so multiple proposals are not counted as multiple control effects.

Schema 1.1 adds bounded feedback without changing identity version 1.0. Readers accept old 1.0 records unchanged, and appending new records upgrades the envelope while retaining historical record versions and bytes. Custom feedback is explicit caller content; library loop guards use fixed safe templates rather than raw input or exception text.

The enclosing intervention comparison-link contract remains version 1.0 and carries each trace's own evidence version. This avoids changing historical comparison payloads merely because a newer reader supports feedback.

The span reference is the existing parent context, not an invented model/tool span for denied work. Use ordinary tracing alongside the harness to capture work spans. Wrap the traced callable inside the harness wrapper to keep policy evaluation outside that callable's model/tool span. Broader host spans and overall elapsed runtime can still include control time.

Run, branch, and evidence references preserve nonempty opaque IDs, including Unicode and paths used by existing applications. Policy IDs, versions, and reason codes remain bounded labels. Identifiers themselves can reveal private information; the harness does not treat them as anonymized data.

Outcomes and timing

Outcome Meaning
proposed A shadow-mode control proposal; execution is unchanged
applied An enforced control action that matches the hook resolution
rejected A control proposal superseded by another policy's stronger action
failed Policy evaluation failed; enforcement can escalate/stop admission
no_op Continue/no policy change, including intrinsic pass-through hooks

An error-triggered escalation remains a failed policy evaluation, not a successful optimization. Multiple policies supporting one resolution share a hook ID. Intrinsic admission decisions use origin harness and do not count as applied user-policy interventions.

Policy evaluation timing includes callback validation/error handling and is separate from shared hook evaluation timing. The latter includes lock wait and evaluation bookkeeping, but excludes evidence serialization. Do not sum the repeated hook_duration_ms across proposals; deduplicate by hook ID as HTML does. Cumulative timings across concurrent hooks can exceed wall-clock duration. They neither create model/tool events nor fabricate usage. Protected calls from inside policies are rejected, avoiding recursive dispatch and duplicate usage; arbitrary unwrapped policy code remains the trusted host's responsibility.

dispatched=false on failed admission distinguishes a pre-dispatch cancellation from cancellation of an attempted call. dispatched=true does not prove that a provider completed work, billed it, or produced side effects. A generator that is never resumed creates no decisions.

Privacy and capture failures

Raw prompts, arguments, results, policy state, and exception messages are not included in decision records. Policy configuration values are also omitted by default. Snapshots retain their hash and declarations, with configuration_capture="redacted" and configuration=null. Hashes identify settings; they do not anonymize secrets or recover omitted settings.

Only set HarnessConfig(capture_policy_configuration=True, ...) for configuration you have reviewed for sharing. This explicitly retains its immutable JSON values. HTML still omits those values unless include_content=True (CLI --include-content) is also selected. These rules do not remove private data that the host independently records elsewhere in the trace.

The agentloop.harness namespace is reserved when capture is enabled. If delivery collides with invalid or conflicting trace metadata, the detached run artifact records a safe trace_evidence_error. Shadow execution continues unchanged; enforce mode stops further admission and fails closed. Original callable failures and cancellation retain precedence over after-hook capture failures. Inspect capture diagnostics rather than assuming trace delivery always succeeded.

Comparisons and persistence

The existing intervention builder and ledger snapshot harness evidence from the actual baseline and candidate traces under metadata["agentloop.harness_evidence"]. This creation-time key is reserved for derived evidence; callers cannot supply a replacement. Old intervention artifacts without it remain readable. No new ledger or database migration is introduced.

The comparison keeps all decision outcomes, but only enforced, applied policy decisions appear in applied_candidate_decision_ids. Shadow proposals, failed policies, and intrinsic admission records are not promoted to applied interventions. Missing retry/conflict targets appear in unresolved_decision_ids; capture errors or unresolved references produce capture_status="known_incomplete". Otherwise it is no_reported_gaps, which does not prove that uninstrumented or unbound work was captured.

comparison_status="observed_pair" means the existing replay compared supplied traces. It does not establish a causal effect for an individual policy; individual_policy_effect remains unverified. Failed quality gates and unknown costs remain in the existing measured report. Original decision and estimator snapshots are retained even if current policy settings or findings later change. Project isolation and idempotent/conflicting writes use the existing SQLite and Postgres store contracts.

Compatibility and export

Native trace schema 1.1 is unchanged: the additive namespace is metadata. Existing traces and ordinary reports remain valid. AgentLoop OTLP exports carry it in the existing agentloop.trace.metadata resource attribute, including empty traces containing only decision evidence. Generic third-party exporters are not assumed to preserve AgentLoop-specific metadata.

Unknown harness-evidence versions remain transportable as opaque trace metadata. The decision reader rejects unsupported/malformed evidence; HTML reports that limitation and omits its raw contents. Interventions refuse invalid source evidence rather than manufacture comparison links. Native metadata is not rewritten or dropped during transport. These artifacts are not a tamper-proof security audit.