Skip to content

A measured Hermes example

On 2026-11-12, the repository-question example ran through real Hermes v0.22.7 (b200f5bc60e31588ba2cb60e288029c4a54ba5fa) and local Qwen2.5-0.5B-Instruct Q4_K_M inference. AgentLoop source was the implementation merged through PR #304. No paid provider requests were made.

The baseline used a model turn to answer and a second turn to repair its format. The proposed change requested JSON immediately and validated its value against the authoritative repository TOML. Independent expected answers were frozen outside the runner. This measured the combined application change.

Comparison Quality-checked pairs Total model calls, baseline → candidate Mean task latency, baseline → candidate Decision
Source-checked JSON candidate 6/6 12 → 6 14.517s → 7.414s Accepted for these tasks
Earlier JSON-only candidate 5/6 See retained per-pair measurements See retained per-pair measurements Rejected: incorrect answer
Deliberately incorrect control 0/6 See retained per-pair measurements See retained per-pair measurements Rejected

The measurement excerpt retains every pair's run IDs, quality results, timings, calls, retries and token provenance, including the earlier failed proposal. It records SHA-256 hashes of the original local reports and the pinned runtime/model assets. It is historical evidence, not a CI fixture or a replacement for raw traces. Model operating cost remains unknown.

Reproduce the workflow using the commands in guided optimization. New runs must retain their own outcomes, including rejections. The local-model CI lane is opt-in; deterministic runtime conformance remains part of ordinary CI.

These were three repository metadata questions with two repetitions on one host and one small model, in fixed baseline-first order. The task timer excluded Hermes startup. There was no instrumentation-overhead control. These results do not establish performance on coding, research, production traffic or other models. The earlier failure demonstrates why format validity alone is an insufficient correctness check.

The source observer accepts Hermes >=0.21.6,<0.23. Contract coverage includes 0.21.6; real-host coverage is pinned to 0.22.7. Hermes 0.23.x support has not been validated. Published AgentLoop 0.1.1 retains its narrower 0.21.x range. See runtime validation for the pinned host and scenarios.