Remaining product and measurement gaps¶
AgentLoop can now guide a developer from initialization through a manually applied change and independently checked before/after evidence. The release work prioritizes this path over adding another control or experiment system.
| Gap | Current evidence | Next useful validation |
|---|---|---|
| First use by someone unfamiliar with the project | Generated offline onboarding and installed-wheel tests | Observe consenting developers connecting existing agents; record time to first useful finding, failed steps and required help |
| Representative agent work | Small Hermes repository questions, older model/tool studies, and a reproducible three-workload guided suite | Collect coding, retrieval and tool-heavy task sets from actual applications, with owners' independent acceptance criteria |
| Recommendation precision | Native evidence/risk rankings, explicit assumptions, retained false positives and rejected changes | Review recommendations against independent labels; report unsupported recommendations and abstentions alongside accepted ones |
| Quality coverage | Frozen task inputs and declared per-task checks; a wrong fast candidate cannot pass those checks | Expand adversarial and held-out tasks; measure quality dimensions beyond exact structured answers |
| Causal performance | Per-attempt measurements, disclosed fixed-order confounds, and an optional interleaved control that alternates condition order | Use interleaved controls in real studies, control cache/load effects further, and increase independent task/host/model coverage |
| Instrumentation overhead | Counterbalanced CPU controls separate recording, JSON export and native analysis at declared span volumes; model controls retain quality failures | Repeat controls around actual application execution, across hosts and realistic payload sizes |
| Cost | Actual token provenance and qualified model-price estimates | Obtain authorized billing data; retain unknown local hardware/energy cost instead of inventing dollars |
| Hermes compatibility | 0.21.6 contract coverage and real 0.22.7/0.23.1 conformance | Revalidate future hook changes against the pinned runtime scenarios before extending the supported range |
| TypeScript adoption | npm package agentloop-profiler, installed Node API interoperability, esbuild/Chromium package checks and a tag-driven npm release workflow |
Observe integration into developer-owned applications and extend the browser/bundler target matrix as needed |
No external users, production deployments or customer savings are established by these tests and studies. PyPI download counts do not identify unique developers or deployments. The retained small experiments support their specific observed outcomes, including failures, without establishing a general optimization rate.
For the next iteration, prioritize the first-use observation and one real application's held-out correctness suite. Those will show whether developers can find a useful inefficiency and prove a safe improvement with less assistance.
First-use developer pilot¶
Use a consenting developer's existing agent and a fresh installed-wheel environment. Have the application owner freeze a small pilot task set and independent acceptance checks before measuring a candidate. Keep held-out tasks separate until the proposed change and quality criteria are fixed.
Record the start time, package/source version, time of first valid trace, time of first actionable finding, failed steps, assistance supplied, and the time the developer reaches an accepted or rejected report. Keep the task manifest, baseline and candidate captures, frozen guided session, linked intervention, paired study and every failed attempt. Record manual effort with the existing study evidence. An optimization rejected by the quality gates is a completed observation with a negative result.
The maintainer must arrange the participant's consent and permission to retain their application data. A sanitized private evidence bundle can use the existing trace/study artifacts. No external participant observations are currently recorded.
Reproduce scoped overhead measurements¶
From the source checkout, replace YOUR_COMMIT with its full commit hash.
uv run --frozen python -m scripts.measure_instrumentation --source-revision YOUR_COMMIT --out runs/instrumentation-controls
The protocol records source hashes, interpreter, host, planned samples and order
before execution. Default sizes are 32, 256 and 1,024 tool spans, with six paired
repetitions per size. Traced-first and untraced-first order alternate. Each pair
checks the same deterministic arithmetic result, validates a native trace and
retains recording, export and analysis timings. Export measures JSON serialization
and a buffered local file write. Failed and cancelled attempts retain their
records. The shared summarize_values helper preserves unavailable samples in
the planned denominator. A fresh output directory protects earlier results.
These CPU controls quantify local operations on their declared fixtures. Real agent inference, external transport, filesystem durability and operating cost need their own controls. Public finding fixtures and owned CPU examples provide regression evidence. Independent developer adoption remains unmeasured.
The retained evidence index contains 118 checksummed files, including both control runs and the existing and corrected finding evaluations. Both control runs completed all 18 planned pairs. Their raw timings, traces and native analyses remain available.
uv run --frozen python -m examples.real_agent_study.archive research/instrumentation-2026-11/index.json --out runs/retained-instrumentation
The final shared-host run's median recording differences were about 1.75 ms for 32 spans, 10.07 ms for 256 spans and 32.96 ms for 1,024 spans. The first run and all individual samples are retained because host load and filesystem latency varied. These values describe this fixture and machine. The same archive retains the schema-repair false finding before its correction and the complete evaluation afterward. The existing initial regression baseline and its labels remain intact.