Skip to content

Hermes runtime pilot

This pilot runs the actual Hermes plugin manager, AIAgent.run_conversation, file tool, and delegated child agent. An OpenAI-compatible server on loopback returns deterministic responses and synthetic usage. Each scenario gets a fresh profile, workspace, session database, and process. It never uses your normal Hermes profile or a real provider credential.

This is integration evidence. It does not measure live model quality, billing, production latency, or optimization gains. Recorded timings are diagnostic only. Auxiliary model calls, gateways, and live provider behavior remain outside the pilot.

Pinned inputs

Input Pin
Hermes v0.22.7, b200f5bc60e31588ba2cb60e288029c4a54ba5fa
Hermes compatibility lane v0.23.1, 3b421578588243f920886e6fe79a6faa87ea026e
Host dependencies Upstream uv.lock, installed frozen without development extras
Host Python Stable CPython 3.14; exact patch recorded in each report
Hermes installer uv 0.13.7, the same version AgentLoop uses
AgentLoop observer Exported from this source checkout; recorder SHA-256 in each report

The runner checks the Hermes revision and tracked source before and after each case. It records the upstream lockfile digest. Generated evidence is marked synthetic: true; qualification adds provenance metadata without changing events, usage, outcomes, timestamps, or durations. The validator checks raw and qualified artifact hashes and compares their contents.

Both pinned hosts run every scenario in CI. The runner and validator accept only these exact revisions, check the version tag against the recorded revision, and reject a collection that mixes host versions. To reproduce the second lane, check out 3b421578588243f920886e6fe79a6faa87ea026e in the commands below and use a fresh evidence directory. This compatibility check does not repeat the historical local-model optimization study or extend its performance results to a new host.

Scenarios and expected evidence

Scenario Evidence checked
Baseline Same user input and final response with observer disabled; no trace
Streaming Real SSE consumption, one model span, synthetic usage of 17 input / 5 output tokens
File tool Actual read_file dispatch and successful fixture content; two model attempts
Retry HTTP 429, failed attempt with unavailable usage, retry event, successful attempt
Interruption Real stream callback and interrupt(hard_cancel=True) from another thread; unfinished request and incomplete capture; replay/CI fail
Tool failure Missing file retains tool error even though the host turn completes; task quality fails
Default privacy capture_content omitted; no model, tool, or final output retained
Delegation Actual child agent reads the file; linked parent/child traces, each with two model calls and 34 / 10 tokens; child usage stays separate

All model responses use the fake model agentloop-local-pilot. Its counts are synthetic provider data, not observed real-model tokenization. Its price remains unknown. Completed host execution and task correctness are evaluated separately; privacy-disabled output cannot serve as task-quality evidence.

Reproduce on Linux or WSL

Use a current AgentLoop source checkout, Python 3.14, Git, and both pinned uv versions as separate executables. Replace the executable paths below with your local installations. Run these commands from the AgentLoop checkout:

/path/to/uv-0.13.7 sync --locked --no-dev --python 3.14
pilot=$(mktemp -d)
.venv/bin/agentloop hermes export-plugin "$pilot/plugin"
git clone https://github.com/NousResearch/hermes-agent.git "$pilot/hermes"
git -C "$pilot/hermes" checkout --detach b200f5bc60e31588ba2cb60e288029c4a54ba5fa
(cd "$pilot/hermes" && /path/to/uv-0.13.7 sync --frozen --no-dev --python 3.14)
for scenario in baseline stream tool retry interrupt tool-error privacy delegation; do
  timeout --signal=TERM --kill-after=10s 180s \
    "$pilot/hermes/.venv/bin/python" -I scripts/smoke_hermes_runtime.py \
    --plugin-dir "$pilot/plugin" --hermes-root "$pilot/hermes" \
    --out "$pilot/evidence/$scenario" --scenario "$scenario"
done
.venv/bin/python scripts/validate_hermes_runtime.py --root "$pilot/evidence"

The harness clears inherited provider/proxy settings before importing Hermes and limits Python socket clients to the owned loopback provider port. This guard is not an operating-system sandbox. Only file/delegation toolsets are enabled, using the owned fixture. Dependency installation requires network access before runtime.

validation.json summarizes all eight cases. Each scenario also has a report, raw native traces under profile/agentloop/traces, qualified copies under native, and HTML generated through AgentLoop's analysis pipeline. The baseline has no trace. The CI workflow runs the same cases and uploads these artifacts, excluding profile databases, caches, and installed dependencies.

This compatibility extension and incomplete-capture replay fix ship in AgentLoop 0.2.0. The immutable 0.1.1 release accepts Hermes >=0.21.6,<0.22. See the integration guide.