Skip to content

Structured workflow quality

Quality contract 2.0 compares routing decisions, labels, bounded numbers, extracted fields and entity pairs. It uses the existing quality, replay, intervention and study APIs. Legacy fixtures without a version keep their existing scorers, result format and behavior.

AgentLoop evaluates caller-defined criteria. A pass applies to those criteria and supplied examples; it does not establish universal correctness or authorize a production decision.

Model-substitution experiments apply these frozen criteria across explicit implementations and export the assessed trials to studies.

Opt in explicitly

Save a versioned fixture suite:

{
  "schema_version": "2.0",
  "fixtures": [
    {
      "id": "route-message-1",
      "expected": "billing",
      "baseline_output": {"route": "support"},
      "candidate_output": {"route": "billing"},
      "input_ref": "task:message-1",
      "expected_ref": "reference:message-1",
      "scorer": {
        "type": "decision",
        "version": "1.0",
        "field": "route",
        "labels": ["billing", "support"]
      }
    }
  ]
}
agentloop quality-report fixtures.json --out quality.md --json-out quality.json
agentloop replay --baseline before.json --candidate after.json --quality-fixtures fixtures.json --min-quality-score 0.9
uv run python examples/structured_quality_workflow.py --out runs/structured-quality

The example creates deterministic synthetic routing, extraction and matching evidence, plus a timeout and an unmatched example. It preserves original predictions, links four intervention records, and generates quality/replay/study JSON and Markdown. Three configured fixture comparisons pass; the timeout cannot win by doing less work. All five baseline tasks remain visible. These are contract tests, not measured application benefits or empirical calibration.

parse_quality_fixtures/load_quality_fixtures copy the wrapper version onto each fixture. Direct Python and HTTP lists declare "schema_version": "2.0" on each fixture. Every fixture in a structured suite must use the same contract and a unique nonempty id. Mixed/unknown versions and unknown configuration fields fail explicitly. New scorer types require the version marker.

Scorers and criteria

Built-in scorer implementation version is 1.0, separate from the 2.0 report contract. Omission selects that built-in version. Trusted custom scorers require an explicit implementation version.

Type Configuration and behavior
decision Nonempty typed labels; exact equality to the reference label. Unknown predictions are explicit failures, not dropped observations.
multilabel Declared label domain and unique list outputs; set F1 with TP/FP/FN counts. Duplicate predictions are invalid. Two empty sets score 1 by the declared empty-set convention.
numeric Finite minimum < maximum, nonnegative tolerance, optional unit; score 1 within the reference tolerance, otherwise 0. Decimal tolerance comparisons use declared values; unrepresentable absolute-error magnitudes remain unavailable.
fields Nonempty expected object; fraction of expected top-level fields with exactly matching typed JSON values. Extra top-level fields are allowed by default; allow_extra: false adds them to the denominator. Nested values compare exactly. This is not full JSON Schema validation.
matches Unique two-element entity-ID pairs; set F1. Pairs are directed unless symmetric: true. Typed IDs remain distinct.
Legacy types exact_match, contains, bounded glob, required_fields, and json_subset can also opt into the new evidence contract. Their configured predicates remain the same.
custom Synchronous trusted local module:function, explicit version, optional JSON configuration; returns a boolean or {passed, score, detail}. HTTP and stored-intervention requests continue to reject custom executable scorers.

Structured types accept an optional field, a literal top-level output-object key. Object/set/matching scorers can read JSON-encoded container strings. Decision labels themselves are not coerced: false, 0, 1, 1.0 and "1" remain typed values. Explicit JSON null is distinct from missing output.

pass_score defaults to 1 and applies to each assessed case. Unknown labels or invalid output shapes cannot pass merely through a lower threshold. A custom scorer's explicit rejection remains a rejection. The suite's min_score also applies to its aggregate. Gates use unrounded scores and exact arithmetic over declared decimal score values; presentation rounding cannot turn a miss into a pass. Users must select meaningful task-specific criteria and thresholds.

Missing data, failures and provenance

Output lookup is explicit: side-specific fixture output, candidate output, or trace.metadata.output. Contract 2.0 does not guess a workflow's final output from its last model span. References are not fetched automatically. Applications that keep trace bodies private can supply evaluation outputs through separate fixture artifacts while retaining only references/hashes in reports.

Known wrong or malformed outputs score zero with an explicit status. Missing outputs/reference outcomes, unknown reference labels, scorer failures and scorer timeouts remain unassessed (score: null). Per-case statuses and side-specific assessed/unavailable denominators remain visible. A side's aggregate score is unavailable unless every case was assessed and execution is complete/unreported; observed_score_mean describes only the available assessments and cannot bypass the gate.

Execution status can be declared per fixture with baseline_status or candidate_status; workflow status is used when available. Known task failure metadata cannot be overridden by a fixture claiming completion. Correct partial output from a failed/cancelled/timed-out run does not make the overall quality gate pass. Standalone output scoring uses unreported execution status and makes no claim about whether an agent actually ran.

Custom exceptions/timeouts and built-in parsing limits are explicit unknown evaluations. Exception bodies and custom details are omitted by default; custom capture_detail: true opts into bounded supplied detail. No hard timeout or sandbox is inserted around trusted Python code. The host must bound scorer work; host cancellation and other fatal exceptions propagate. Inputs/configuration are copied so one scorer invocation cannot accidentally mutate the next one's data.

Reports preserve contract/scorer versions, implementation references, configuration hashes, expected/output hashes, reference IDs and source trace IDs. They omit raw input/output bodies and raw scorer configuration by default. Keep the original fixture/configuration artifacts alongside a report for reproduction. Hashes identify evidence; they do not authenticate a grader, prove independence, or anonymize predictable labels.

Classification counts

decision reports optionally include confusion-derived metrics grouped by scorer configuration. Matrices are sparse [expected_index, predicted_index, count] triples with matrix_format: "sparse_row_column_count"; the final prediction column represents invalid/out-of-domain predictions. Storage grows with labels and observed pairs rather than the square of the declared label count.

Accuracy/precision/recall/F1 describe available labelled outputs. Missing outputs, unlabelled cases, invalid predictions, unassessed outputs and coverage remain explicit. Missing references and outputs can overlap. These conditional metrics are not substitutes for end-to-end task success; an incomplete suite still cannot pass. Other scorer families do not receive an imposed confusion matrix.

Attach results for studies

from agentloop import attach_quality_report
from agentloop.quality import build_quality_report, parse_quality_fixtures

# suite is a caller-owned {"schema_version": "2.0", "fixtures": [...]} object.
fixtures = parse_quality_fixtures(suite)
quality = build_quality_report(
    fixtures, baseline_trace=baseline, candidate_trace=candidate, min_score=0.9
)
attach_quality_report(baseline, quality, side="baseline")
attach_quality_report(candidate, quality, side="candidate")

Attachment writes versioned agentloop.quality metadata without overwriting host metadata. It validates report identity/hash and any trace binding. Attach before saving the final traces and creating ledger records, so their fingerprints include the selected evidence. A report produced without source trace IDs is an explicit caller-provided association when attached later.

Replay uses the new evidence, leaves unknown score/delta values null, and adds fail-closed quality gates. It does not fall back to an older scalar score when attached evidence is invalid or incomplete. Original output predicates and unversioned quality behavior stay intact. Intervention records preserve the quality report without changing prediction snapshots; deterministic results keep same-input ledger creation idempotent.

Existing paired studies consume attached side evidence. They retain unmatched, failed and unscored runs and export scorer provenance alongside quality metrics. Versioned reports also expose recorded classifier/rule decision-span counts and error-span counts with existing model/tool/resource metrics. These count recorded spans, not all possible hidden decisions or unique business outcomes. Caller pairing keys still determine experiment identity; repeated tasks need appropriate task-aware uncertainty analysis, as described in studies and ablations.