Skip to content

CI self-tests and application comparisons

The repository's AgentLoop replay self-test generates synthetic demo traces and verifies replay/quality gating and report generation. Passing it confirms the machinery works on those fixtures. It does not demonstrate that an arbitrary PR improves AgentLoop or an application's performance. The existing required-check name, Replay and optimization gates, is retained for branch-policy compatibility.

CI JSON and Markdown identify each input's run ID, path when available, producer source, and synthetic marker. A missing marker is reported as missing rather than as proof that the data came from production. Comparisons containing synthetic data are explicitly qualified. Demo generators continue to mark their traces.

Hosted container pulls

Hosted Python/Postgres and deployment checks pull Docker's Official Images from ECR Public (public.ecr.aws/docker/library) to avoid shared unauthenticated Docker Hub pull quotas. CI keeps the postgres:16-alpine service/version and the source Dockerfile's Python tag. Deployment uses a CI-only Compose image override and a temporary Dockerfile with only the registry prefix replaced. Production Dockerfile/Compose defaults, health checks, database contracts and application validation remain unchanged.

Docker lists Python/Postgres in its official ECR publisher gallery. These mirrors do not replace checks: pull failures still fail CI, and the full interpreter, Postgres and deployment tests must run before a PR can merge.

Compare application traces

Run your application's baseline and candidate, then upload the resulting trace files as an artifact in the same workflow run. The reusable workflow downloads that artifact and runs the configured gates. It never generates replacement demos: missing artifacts, missing trace files, malformed suites, and failed gates fail the job. Reports remain available as artifacts and in the job summary.

jobs:
  record:
    runs-on: ubuntu-latest
    steps:
      # Check out/install your application and run its baseline/candidate here.
      # Produce baseline.json, candidate.json, and optional quality.json in traces/.
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
        with:
          name: application-traces
          path: traces/
          if-no-files-found: error
  compare:
    needs: record
    uses: dipeshbabu/agentloop/.github/workflows/application-performance.yml@main
    permissions:
      contents: read
    with:
      trace_artifact: application-traces
      baseline_trace: baseline.json
      candidate_trace: candidate.json
      quality_fixtures: quality.json
      min_latency_improvement_pct: 10
      min_cost_improvement_pct: 5
      min_quality_score: "0.9"
      agentloop_ref: main

Replace the comments with your application's setup and tracing commands. Omit quality_fixtures and min_quality_score when no quality suite is provided. All candidate cases must pass when a suite is supplied. Threshold defaults for latency/cost improvement are zero; an unevaluable positive cost target fails. Use distinct report_artifact names when comparing multiple pairs in one run.

For repeatable CI, pin both the reusable workflow reference and agentloop_ref to the same reviewed AgentLoop commit. The workflow checks out AgentLoop in its own directory and installs its locked core dependencies. It does not install your application or SDKs. Use built-in quality scorers here; custom Python scorers that need application modules belong in your application's own job using agentloop ci.

The repository smoke workflow exercises artifact upload/download, the reusable workflow, and quality gates using explicitly synthetic fixtures. These smoke results carry the same synthetic-data qualification.

Manual repository runs

The existing agentloop-performance.yml workflow also accepts checked-in trace paths through workflow_dispatch. Set generate_demo_traces to the string false to use those files without generating demos. They must exist at the selected revision; failed file checks never fall back to demo data.