Offline semantic judgments¶
AgentLoop's judgment contract 1.0 provides optional, provider-independent
answers to explicitly posed questions about selected execution spans. Ordinary
trace.report(), diagnosis, HTML, replay and store operations never execute a
judge. They can display already attached receipts. No provider SDK, credentials
or remote service is required.
Use the adapter and benchmark layer to compare rules, local predictors and optional typed-service transports on frozen labelled cases. Use semantic waste investigations to bind optional judgments to canonical finding rules and explicit task criteria.
Judgments are separate from observed execution facts, deterministic finding rules, and caller-defined task quality. A judgment is neither universal correctness evidence nor permission to change production behavior. Judgment execution alone does not create semantic waste findings or automatically substitute models.
Python interface¶
from agentloop import (
JudgeIdentity, JudgeUsage, JudgmentAnswer, JudgmentSession, JudgmentSpec,
LocalCallbackJudge, attach_judgment, judgment_request,
)
def local_rule(request, *, timeout_s):
# Only explicit references, numeric/status observations and caller-supplied
# summaries are available here. This illustrative rule is not task quality.
summaries = [item.summary for item in request.evidence]
if any(item is None for item in summaries):
return JudgmentAnswer(status="unknown", reason="insufficient_evidence")
return JudgmentAnswer(
summaries[0] == summaries[1],
usage=JudgeUsage(0, 0, 0.0, "reported", "reported"),
)
identity = JudgeIdentity.configured(
"example.summary-equality", "1", {"comparison": "exact"},
provider="local", model_or_rule="summary-equality", revision="1",
)
session = JudgmentSession(LocalCallbackJudge(identity, local_rule), cache_size=128)
request = judgment_request(
trace, JudgmentSpec("Do these approved summaries match?", "boolean"),
["first-span", "second-span"],
summaries={"first-span": "Approved summary", "second-span": "Approved summary"},
)
receipt = session.evaluate(request, enabled=True, timeout_s=1.0)
attach_judgment(trace, receipt)
trace.export_json("judged-trace.json")
JudgmentBackend is a Python protocol with a JudgeIdentity property and
judge(request: JudgmentRequest, *, timeout_s: float | None) -> JudgmentAnswer.
Adapters implement this protocol; analysis consumes the AgentLoop contract.
LocalCallbackJudge accepts a trusted synchronous callable directly, without
dynamic imports or executable paths from trace metadata. It forwards timeout_s
as a keyword argument. Asynchronous callbacks/results are not supported.
The immutable request contains a run ID, a JudgmentSpec, and a tuple of
JudgmentEvidence. The four answer types are:
| Kind | Required declaration | Valid known answer |
|---|---|---|
boolean |
Question | Exactly True or False; integers are rejected |
probability |
Question | Finite numeric value in [0, 1], excluding booleans |
choice |
Question and unique nonempty string choices |
Exactly one declared string |
score |
Question and finite minimum < maximum |
Finite numeric value within inclusive bounds |
Probabilities and confidence values are uncalibrated unless the backend supplies
JudgmentUncertainty(calibration_status="measured", calibration_ref=...).
The reference must point to caller-owned calibration evidence; AgentLoop records
this assertion, without verifying that the measurement is valid for a new task.
Confidence, when supplied, must be in [0, 1] and is distinct from the answer.
Failure, latency and usage semantics¶
Execution defaults to enabled=False, producing a disabled receipt without
dispatch, even if a cached answer exists. Explicit execution returns known,
unknown, timeout, error, or invalid_result. Unknown answers require
reason="abstained" or "insufficient_evidence"; failure results carry no answer.
Out-of-domain values, wrong types and changing backend identity during a call
cannot become a known result. Exceptions expose only bounded public categories,
never exception messages or arbitrary class names. Fatal cancellation and other
BaseException subclasses propagate.
Timeouts are cooperative. The backend owns cancellation and resource limits.
AgentLoop passes a time budget, recognizes TimeoutError, and discards an answer
that arrives after that budget, retaining reported usage. A synchronous callback
that never returns is not forcibly interrupted. Run untrusted or unbounded work
in a host-owned isolated process. No background worker or hidden retry is started.
JudgeUsage preserves input/output counts independently, with token_basis
reported, estimated, or unknown. Cost in USD has a separate cost_basis
reported, calculated, or unknown. Missing usage is None, including failures
where a provider may already have charged. The backend must explicitly report
zero for a free local rule. Declared usage and its basis must agree.
The original evaluation's ID, timestamp, latency, usage and uncertainty remain
intact on cache hits. Each invocation receives its own ID, dispatch/cache flags,
latency and incremental usage. Cache hits and disabled execution carry zero
incremental model usage with basis not_dispatched. Read summaries sum invocation
costs, not repeated copies of cached evaluation costs. These are offline analysis
costs and timings; workflow cost, runtime, token totals and quality stay separate.
Evidence and privacy¶
judgment_request selects unique existing span IDs. Its automatic projection
contains only ID, source hash, normalized operation kind, normalized status and
duration. It does not forward trace/event names, prompts, outputs, tool payloads,
errors, arbitrary metadata, provider configuration or model names. Unknown
operation/status labels normalize to unknown. Caller-authored questions,
identifiers, judge identities and explicit derived summaries are visible to the
backend and exported receipts; use permission-cleared values for them. References
are opaque strings, never automatically fetched URLs or file paths.
The source hash binds the normalized selected event, including its locally held
content and metadata. Hashing does not redact the original trace and is not a
privacy guarantee for low-entropy data. Transport-only otel_span_id and
otel_trace_id are excluded. Changes to selected source content invalidate
evidence even when the automatic projection is unchanged. Context outside the
selected spans is not silently added; include needed context in an approved
summary and update it when that context changes.
attach_judgment copies and validates a receipt and checks its source binding.
It preserves other trace metadata and earlier receipts. Reattaching the same
invocation is idempotent; conflicting or duplicated invocation records are
invalid. read_judgments retains original receipts but reports effective_status
stale and effective_value=None if selected source evidence no longer matches.
Consumers must use these effective fields, not an archived answer in isolation.
Malformed or unsupported evidence is explicit and cannot trigger execution.
The semantic_judgments.status field describes these optional receipts;
the existing analysis_complete field continues to describe deterministic
finding-rule execution.
Cache identity and storage contract¶
Each session has a bounded LRU of successful results only, default capacity 128.
Use cache_size=0 to disable it and clear_cache() to invalidate it. A session
belongs to one backend and one analysis worker; it is not thread-safe. Caches are
not shared across sessions/backend instances and are never reconstructed from
trace metadata. Repeated source validation hashes each selected event once per
read, even when many receipts refer to it.
The cache key is SHA-256 over canonical finite JSON containing the contract version, full typed request/evidence, and full judge identity. Identity includes implementation, version, provider, model/rule, revision and configuration hash. Questions, choices, bounds, summary text, source content, configuration, revision and implementation changes therefore change the key. Identity/version/configuration are adapter declarations: adapters must bump them when behavior changes. A mutable callback closure or external model alias cannot be automatically detected; use a new session/revision, or disable caching for such a backend. The configuration helper hashes configuration without retaining it or configuring the callback.
Native trace schema 1.1 remains unchanged. Metadata key agentloop.judgments is:
Each record contains these required contract fields:
| Field | Meaning |
|---|---|
schema_version, evidence_kind |
1.0, semantic_judgment |
request |
run_id, typed spec, ordered evidence objects |
judge |
The six JudgeIdentity fields |
cache_key |
Hash of version, request and judge |
evaluation |
id, evaluated_at, status, value, reason, usage, uncertainty, latency_ms |
invocation |
id, cache_hit, dispatched, latency_ms, incremental usage |
record_hash |
SHA-256 of the entire record excluding this field |
Canonical JSON uses sorted object keys, compact separators and finite JSON
values. The Python dataclasses document field names and validate primitive
constraints; validate_judgment_record validates the receipt relationships and
hashes. Hashes detect alteration and binding errors, not author authenticity or
truthfulness. Unsupported contract versions remain stored but are not interpreted.
Native JSON, SQLite and OTLP preserve the metadata and provenance. If a transport removes original source content, the archived receipt remains exportable but its effective answer becomes stale on the redacted trace. This deliberately avoids accepting a judgment against evidence that is no longer present.
Run the local synthetic example with
python examples/semantic_judgments.py --out runs/semantic-judgments.
It requires no provider credentials and is not empirical optimization evidence.