Claude Code¶
AgentLoop can profile and compare Claude Code
runs through the guided workflow. The
runner
runs claude -p --output-format stream-json --verbose for each task and
converts its event stream into a native AgentLoop trace. Claude Code is not
modified and no plugin is installed.
AgentLoop 0.2.6 and newer install the runner with the package: run it as
python -m agentloop.runners.claude_code, using the Python environment where
AgentLoop is installed. In 0.2.5 it is only in a repository checkout, as
python examples/claude_code_runner.py.
Each task attempt is a real Claude Code run on your Claude plan or API key and counts against that usage.
How events become a trace¶
The runner starts Claude Code in the directory given by --cd, sends the task
prompt on stdin and pre-approves only the read-only Read, Grep and Glob
tools with --allowedTools. A request for any other tool is denied in
non-interactive mode and recorded as a failed tool call. Stream events carry no
timestamps, so the runner timestamps each JSON line as it arrives.
| Claude Code event | AgentLoop event |
|---|---|
assistant events sharing one message.id |
One model_call named claude_response, from the session start or the last tool result until the last part of that response arrived |
tool_use block |
tool_call named after the tool, ending when its tool_result arrives; is_error results are errors |
message.usage |
Provider token counts with usage_scope: request. Input tokens include cache reads and writes, which are also listed in metadata |
result |
The answer scored by the frozen task checks. Turns, durations and reported cost are kept on the root claude_code span |
result with is_error or a non-success subtype |
Error on the root span |
Claude Code may emit each content block of a response as its own assistant
event with repeated usage. The runner counts each response once.
Run the guided workflow¶
Tasks use a prompt string or an object with prompt. The Codex
task set
works unchanged: it asks three metadata questions about this repository with
exact expected values.
improve compare (AgentLoop 0.2.5 and newer) compares an unchanged and a
changed configuration in one command. Arguments after a second -- go to claude unchanged, so the
candidate below only lowers the turn limit:
agentloop improve compare --tasks examples/codex_repository_tasks.json --out runs/claude --timeout 600 --change "Limit lookups to four turns" --before-arg python --before-arg -m --before-arg agentloop.runners.claude_code --before-arg --cd --before-arg . -- python -m agentloop.runners.claude_code --cd . -- --max-turns 4
Runner options: --cd sets the directory Claude Code works in,
--instructions FILE prepends shared instructions to each prompt, --model
passes --model and records the label, --condition labels the trace, and
--claude selects the executable.
The conversion is tested against the documented stream shape and an end-to-end run through a stand-in executable. A recorded run against a live Claude Code installation has not yet been published here.