OpenAI Codex CLI¶
AgentLoop can profile and compare Codex CLI
runs through the guided workflow. The example
runner
runs codex exec --json for each task and converts its event stream into a
native AgentLoop trace. Codex is not modified and no plugin is installed.
Each task attempt is a real Codex run on your ChatGPT plan or API key and counts against that usage.
How events become a trace¶
The runner starts codex exec --json --ephemeral --skip-git-repo-check
--sandbox read-only --cd DIR - and sends the task prompt on stdin. Codex events
carry no timestamps, so the runner timestamps each JSONL line as it arrives.
| Codex event | AgentLoop event |
|---|---|
command_execution, file_change, mcp_tool_call, web_search items |
tool_call spanning item.started to item.completed; non-zero exit codes are errors |
agent_message and reasoning items |
model_call ending when the item arrived |
| Time before a tool call with no visible item | model_call named codex_tool_decision |
turn.completed usage |
Provider token counts on the turn's last model span, with usage_scope: turn_total, cached and reasoning tokens in metadata |
turn.failed or error |
Error on the root codex_exec span |
Codex reports usage once per turn, so per-span token counts other than the turn total are recorded as unavailable instead of being estimated. The final agent message is the answer scored by the frozen task checks.
Run the guided workflow¶
Tasks use a prompt string or an object with prompt. The checked-in
task set
asks three metadata questions about this repository with exact expected values.
agentloop improve capture --tasks examples/codex_repository_tasks.json --out runs/codex/baseline --timeout 600 -- python examples/codex_exec_runner.py --cd .
agentloop improve start --tasks examples/codex_repository_tasks.json --baseline runs/codex/baseline/capture.json --out runs/codex/session
Arguments after a second -- go to codex exec unchanged. A candidate can
therefore change the model, reasoning effort, profile or features without
editing the runner. Capture it with the unchanged runner as an
interleaved control:
agentloop improve capture --tasks examples/codex_repository_tasks.json --out runs/codex/paired --timeout 600 --control-arg python --control-arg examples/codex_exec_runner.py --control-arg --cd --control-arg . -- python examples/codex_exec_runner.py --cd . -- -c model_reasoning_effort="low"
agentloop improve finish --session runs/codex/session/session.json --candidate runs/codex/paired/capture.json --change "Lower reasoning effort for lookups" --out runs/codex/results
Runner options: --cd sets the directory Codex reads, --instructions FILE
prepends shared instructions to each prompt, --model passes --model and
records the label, --condition labels the trace, and --codex selects the
executable.
First observed run¶
On 2026-11-14 the maintainer ran this task set with Codex CLI 0.167.1 on Windows, using the configured model at its configured reasoning effort. Each of the three tasks ran twice per condition.
- Baseline: all six answers were correct. Attempts took 15.6–29.8 seconds
and used 38,755–39,901 input tokens to read one field. AgentLoop reported
split_large_stepon every attempt, pointing at the size of the model context. - Candidate: disabled the unused
apps,browser_useandcomputer_usefeatures to shrink the tool context. It ran against an interleaved control, with order alternating between attempts. - Outcome: all twelve answers were correct, and four of six pairs passed.
Input tokens fell by about 2,600 (about 7%) in pairs where both runs took the
same path. Both
engines.nodepairs were slower than their controls, by 12% and 19%, and failed the latency gates. The change was rejected.
Three runs, two controls and one candidate, used about 18,000–20,400 more input tokens than their other repetition. Each of them took an extra model step. Codex ran one tool, called the model again, and then ran another tool, instead of running both tools in one step, and each model step resent about 19,000 tokens of context. Several runs also read a user-level instruction file referenced by the global Codex instructions. That read cost nothing extra when Codex ran it in the same step as the requested file read. The extra step, not the read itself, accounted for the increase. This path variation is larger than the measured context saving.
The repetition_detour finding, available from AgentLoop 0.2.3, reports these
attempts automatically. On these 18 traces it flags exactly the three runs with
an extra model step and none of the others. More repetitions would be needed before
attributing a latency change to this configuration. These six pairs describe
one machine, one model configuration and three lookup tasks.
Second observed run¶
On 2026-11-17 the maintainer repeated the task set with four repetitions per
task, using the runner's model-round accounting, an interleaved control and a
frozen --decision aggregate.
- Baseline: all twelve answers were correct.
repetition_detourflagged one attempt that took an extra model step and used 20,035 more input tokens than its typical repetition. One other attempt also took an extra round, but it added only about 480 input tokens and was not reported. - Candidate:
--instructionsasked Codex to request every file read it needed together, in one step, and to read no other files. It ran against the unchanged runner as the interleaved control. - Outcome: all 24 answers were correct, and the candidate took no costly detours, while the control took one. But the candidate was slower in all 12 pairs, with a median latency change of −34.5%, so the aggregate decision rejected it. Its first model round spent noticeably longer planning, and it averaged 2.42 model rounds against 2.08 for the control. It still read the user-level instruction file, though in parallel with the requested read.
Removing occasional detours did not pay for the extra planning time on these lookup tasks. Like the first run, this describes one machine, one model configuration and three tasks.