Skip to content

Guarded harness implementation and evidence review

Reviewed on 2026-09-30 for roadmap #182, through PR #249. The core controls, empirical evaluation and deferred follow-ups are implemented. These APIs and the evidence workflow used by the real-agent pilot are now included in release 0.1.0. The review records implementation evidence; it does not activate controls in production.

Implementation coverage

Roadmap item and merged PR Supported scope and evidence
#183 — #221 Opt-in Python lifecycle controls, explicit capabilities and disabled/shadow/enforce modes; lifecycle tests.
#184 — #222 Native decisions and intervention links, configuration hashes, actual resolution and overhead; evidence tests.
#185 — #223 Atomic admission budgets, reservations, usage provenance and concurrent accounting; budget tests.
#186 — #224 Bounded retries and caller-declared progress, explicit side-effect safety; loop tests.
#187 — #225 One pinned LangGraph adapter; shared enforcement and real SDK tests.
#188 — #226, #242 Frozen ablation protocols and a real SQL-agent ablation, with original observations and independent quality.
#189 — #227, #243 Offline estimator calibration and empirical calibration, including abstentions, failures and held-out diagnostics.
#190 — #244 Explicit context transformations, protected results and shared summary budgets; paired context study.
#191 — #245 Allowlisted model routing, capability checks and bounded fallback; routing study.
#192 — #246 Declared tool scheduling, bounded concurrency and retained partial work; SQL scheduling study.
#193 — #247 Versioned completion checks, bounded repair, deadlines and untrusted feedback handling; completion tests.
#194 — #248 Narrow host checkpoint recovery, atomic claims, lineage and carried quotas; protocol tests and pinned SDK tests.
#195 — #249 Reviewed budget-policy promotion, explicit opt-in, bounded canaries, version invalidation and rollback; promotion tests.

Roadmap gates

Gate Review result
Ordinary tracing remains compatible Harnesses default to disabled. Existing integration conformance, telemetry conformance, context and trace-schema suites remain in CI.
Denied work does not dispatch Shared custom-Python/LangGraph conformance covers denial, cancellation and lifecycle handling. Each advanced adapter validates its declared coverage; unsupported paths fail explicitly.
Limits have accurate guarantees Budgets, scheduling, completion and recovery document cooperative admission, unknown usage and already-running work. There is no process sandbox, forced thread termination, provider billing cap or exactly-once side-effect guarantee.
Actions preserve provenance Native decision records retain policy/config versions, mode, resolution, call linkage and evaluation timing. Adapter receipts and audit exports retain failed attempts. Raw arguments/outputs are excluded by default; explicitly supplied identifiers and content hashes still need appropriate handling.
Evaluation preserves losses and uncertainty Frozen plans, original predictions, held-out tasks and source-linked artifacts retain failures, early stops, missing pairs, quality losses and unknown cost. Scoring remains independent of a policy's own completion checks.
Activation is reviewed and reversible The supported promotion family requires eligible empirical evidence, authenticated host/operator approval, explicit scope and a bounded canary. Kill switches and regression/expiry/version checks restore the reviewed baseline. No studied candidate is activated by this review.
Changes pass repository validation Implementation PRs include tests and compatibility notes. Final #249 checks passed after correcting an existing synthetic timing fixture. A skipped CodeRabbit review was recorded as skipped, not counted as a completed review.

What the measurements establish

The real-agent pilot and routing follow-up retain faster smaller-model candidates that lose answer quality. The harness ablation retains early stops as failed tasks. Context summarization reduced main-request input while adding total tokens and latency. Tool scheduling achieved declared concurrency and beat its shadow wrapper while retaining overhead against tracing. Calibration preserves unavailable attribution and unknown operating cost rather than changing runtime coefficients.

These are public, permission-cleared local workloads with retained negative results, not evidence of universal benefit or production-traffic performance. Zero paid-provider spend does not make operating cost known. Each study page links its frozen protocol, checksummed archive and reproduction instructions. Quality, latency, token use, provider spend and whole-workload operating cost keep their distinct meanings.

Validation snapshot

At the initial #195 implementation head, the isolated Windows suite passed 1,842 tests with 82 skips; the affected suites passed 238 tests on both Python 3.10 and 3.13. PR CI exposed a pre-existing routing fake that invented more provider time than a fast host request could take. The final test-only correction adds deterministic zero/sub-millisecond cases; 49 routing/promotion tests and all pre-commit checks passed locally. Production study code and archived evidence were unchanged.

All 17 final PR checks passed on e9a06bf612a56e4e176266e1c08126d1486a94cc, including Python 3.10/3.13/3.14, PostgreSQL-backed tests, the pinned LangGraph suite and CLI workflows, wheel smoke, Docker deployment smoke, standalone Linux/Windows/macOS builds, CodeQL, dependency review and replay/provenance checks. The final CI run records the platform and optional-dependency coverage. Local skips are not claims that unavailable integrations were exercised on Windows.

Further ranking, experiments, scale, onboarding and drift work is tracked by roadmap #208.