Skip to content

Evidence-aware investigation ranking

Included in 0.1.0. Ranking helps select experiments; it never applies a finding or changes an estimated saving into a measured result. Canonical finding IDs, estimator snapshots, predictions and lifecycle decisions remain intact. Severity is a separate diagnostic field.

Every diagnosis and stored queue item includes a versioned ranking object with components, units, input references, missing/invalid reasons and ordering rules. Default ordering compares, in sequence: test readiness, declared confidence, evidence strength, lower quality risk, observed run frequency, cumulative recorded span work, lower validation effort/cost, estimated latency/cost/token opportunity, then stable identity. Confidence/evidence tiers are ordinal categories, not calibrated probabilities. There is no universal ROI formula or dollar/time exchange rate. A moderate supported opportunity can precede a large speculative estimate.

Readiness and explicit inputs

ready_to_test requires complete span evidence, medium/high rule confidence with rule provenance, declared/observed evidence or deterministic inference, a known cost opportunity, some positive modeled opportunity, and complete required inputs. Inferred/semantic-judge evidence alone remains qualified. High declared quality risk, irreversibility, invalid inputs or approximate overlap selection require further evidence/review. needs_evidence_or_review retains the finding and all its original estimates; it does not mean the potential opportunity is zero.

Quality risk, reversibility, validation effort and validation cost have no default guesses based on finding type. Supply them through trace metadata or the optional ranking_inputs argument to build_diagnosis/rank_findings:

trace.metadata["agentloop.ranking_inputs"] = {
    "parallelize_tools": {  # Rule ID; a finding-ID entry can override individual inputs.
        "quality_risk": {"value": "medium", "kind": "declared", "source_ref": "criteria:v2"},
        "reversible": {"value": True, "kind": "declared", "source_ref": "rollback:v1"},
        "validation_effort_minutes": {"value": 15, "kind": "declared", "source_ref": "experiment-plan:v1"},
        "validation_cost_usd": {"value": 0.25, "kind": "declared", "source_ref": "experiment-plan:v1"},
    }
}

Each input requires an explicit value, evidence kind and nonempty source reference. These are trusted host declarations for review, not authenticated proof of safe behavior. Values and references are owned copies. Unknown extra payloads are not copied into ranking input snapshots. Invalid known inputs remain explicit and cannot become eligible after persistence.

Optional inputs are evidence_level (a declared evidence assessment) and estimated_latency_savings_ms, estimated_cost_savings_usd, or estimated_token_savings (all require kind estimated). A supplied opportunity estimate affects ranking only; the original finding prediction remains unchanged and the alternate source reference is visible. Unmodeled or unknown cost is not silently treated as a known zero. A zero estimate is accepted only when explicitly modeled or declared with provenance.

Rule authors can add bounded names to FindingRule(..., ranking_requirements=(...)). The core checks presence and provenance of these scalar inputs without knowing domain semantics. For example, a rule can require validation_dataset; its value and reference must then be provided before that finding is ready to test. The standard requirements cannot be removed by a rule.

Individual dimensions and surfaces

Supported sorts are priority, impact, frequency, latency, cost, tokens, confidence, evidence, quality_risk, reversible, validation_effort and validation_cost. Unknown values sort last. Dimension sorting never changes an item's readiness or grants approval.

agentloop diagnose --path trace.json --sort-by cost --json-out diagnosis.json
agentloop analyze trace.json --sort-by latency --json-out analysis.json --html analysis.html
agentloop optimization-queue --sort-by validation_effort --json-out queue.json
uv run python -m examples.finding_ranking --out runs/finding-ranking

The offline example uses synthetic declarations and performs no optimization. HTML is sorted at generation time and retains its restrictive content security policy: source content never becomes executable markup. Its ranking details show components/references beside the existing estimator provenance. The dashboard offers the same dimensions and exposes readiness and inputs.

GET/POST /v1/traces/{run_id}/diagnosis and GET /v1/optimization-queue accept ?sort_by=...; legacy route aliases retain the same behavior. GET diagnosis remains read-only. POST preserves human lifecycle decisions while saving the new ranking metadata. Existing finding-list cursor pagination is unchanged. JSON preserves both original estimates and ranking inputs/provenance.

Queue accounting and compatibility

Queues retain compatible-span savings selection per project/run. Overlapping findings do not add the same opportunity twice. Ranking reuses the existing selection when its inputs match, recomputing only for different declared estimates. Approximate selection is reported explicitly. The most cautious member confidence, evidence and risk determine group readiness; a well-supported member cannot hide another member's missing inputs. Member records preserve differing declarations.

Observed impact is cumulative duration of distinct recorded spans, which may be inclusive or concurrent. It is not elapsed runtime saved. Frequency counts included distinct project/run pairs, not a population rate. The stored queue's existing collection limit still bounds its sample. Equal run IDs in different projects stay distinct, with affected_run_refs identifying their scope. Overlapping token estimates are retained per member rather than summed without a joint model.

The compatibility field priority_score is now an ordinal among ready investigations; zero marks items needing evidence/review. Its former weighted severity/savings formula is retired. priority_rank records the default order even when a different dimension is displayed. Unknown risk remains unknown, requires_scorer stays true, and safe_to_auto_patch stays false. Ranking provides no automatic application permission. No storage migration or estimator coefficient change is introduced.

See finding rules, the trust benchmark, the generic experiment contract, and reviewed policy promotion for separate detection, validation and deployment contracts.