Skip to content

Offline OTLP JSONL import

The importer included in 0.1.0 reads bounded OTLP JSONL exports and creates qualified native traces, immutable source receipts and a record inventory. It accepts multiple traces on one line and joins a trace split over several lines using the existing agentloop.otel.traces_from_otel parser.

agentloop telemetry import-jsonl harbor-traces.jsonl --source harbor --out runs/harbor-otel
agentloop telemetry import-jsonl trace.json --single-json --out runs/otel-json
uv run --frozen python examples/otlp_jsonl.py --out runs/otlp-example

No Harbor, Omnigent, OpenTelemetry SDK, collector or provider runs during import. The older agentloop import-otel command keeps its single-document behavior. The example uses owned synthetic data, including frozen outputs from an actual pinned Harbor converter. It establishes no live interoperability or measured optimization benefit.

Supported records

Records contain resourceSpans and/or resourceLogs in OTLP JSON format. The existing parser's flat spans array and attribute-object convenience form are also supported. Trace/span identifiers must be nonzero hexadecimal IDs; camel case and the existing snake case aliases are supported. Contradictory aliases, duplicate attribute keys, ambiguous AnyValues and invalid structural types fail with payload-free diagnostics.

Missing IDs use explicitly calculated, source-record-scoped identities. They do not merge across records, deduplicate by appearance or establish a measured session runtime. Actual trace IDs remain in receipts/event metadata; native run and event IDs are namespaced by source and supplied job/trial/task context. Missing session identifiers and ambiguous sessions stay explicit. Caller native IDs and parent assertions do not override the source span identities.

The importer keeps raw span/log counts separate from deduplicated span counts. Only an identical span ID within an identified trace, with identical canonical source span/resource/scope content, is deduplicated. Raw payload fingerprints are compared before minimization: different omitted text still conflicts. Same-looking client/server spans with different IDs remain separate observations.

A conflicting span segment retains the first projection and emits a source/line diagnostic. It never overwrites on last write. Precise trace latency and complete usage become unavailable for the conflicting group. Parent edges resolve only within the same source trace, including parents arriving later. Unresolved or cyclic parentage remains recorded; cycles do not become native graph edges. Span links stay metadata relationships, without fabricated parent edges.

Late external evaluation logs remain in receipts even when attached to a duplicate span segment. Their scorer names, dimensions and scales are supplied by the source; they do not establish an independently verified task pass. A log-only or empty record does not manufacture a successful execution trace.

Measurements and provenance

Reports use existing external qualifications. Missing bounds, source status, usage or cost remain null/unknown. A span without operation semantics stays an unknown source span; it does not manufacture model inference from a name. Model-block usage is separate from nonmodel root aggregates. Cached input is a prompt subset. Recorded aggregate call multiplicity remains separate from native model-block count and never creates independently timed calls.

Source token provenance is retained in source_token_provenance; imported counts remain externally reported rather than being promoted into exact provider measurement. Recorded source model cost remains event/receipt metadata. The batch importer does not apply a price table. Exported spans alone do not prove execution completeness, so replay/study/value improvement claims require the separate trial/verifier path.

Harbor converter boundary

Five checked-in JSONL files were produced by the actual reviewed converter modules at Harbor d5ac1be17f575852eaf4fffc4072fd18481c209b, using owned ATIF v1.7/v1.8, deterministic, aggregated and embedded-subagent inputs. Converter provenance pins input/output bytes, source file hashes, Python, protobuf and opentelemetry-proto==1.42.1. The upstream package metadata says 0.1.1; emitted SDK/scope version is 0.1.0. These are distinct recorded facts.

The pinned converter derives model span endings from neighboring step timestamps and assigns synthetic tool offsets/durations. It emits OK statuses independently of a verifier. harbor-atif2otel resource/scope identity therefore triggers timing_provenance: inferred, preserves its source intervals, and withholds measured duration/runtime and successful source status. These output values must not be presented as measured zero-latency operations or verified success.

Actual v1.8 image/audio references are retained by the converter in text attributes without binary reads. The default batch import minimizes that text. Direct ATIF import retains additional interaction/history, copied-context, source timestamp, media and document relationship semantics; OTLP receipts identify this conversion loss. Converter-generated traces are not proof of either upstream runtime execution or task correctness.

For an existing Harbor export, supply --source harbor; do not install or run Harbor merely to analyze a saved file. This importer does not modify upstream export filtering, scheduling, locks or benchmark rewards.

Bounds, invalid records and output

The shared limits apply to JSON bytes/depth/nodes, total bytes, physical lines, source telemetry items, trace groups, per-trace events, links and source metadata. Reading uses bounded readline; an oversized line is drained in bounded chunks. It remains one invalid record and all bytes still consume the total budget. Blank lines also consume physical record/byte budgets. Overall byte/record/span/trace/reference and merged event limits are terminal.

By default, independently malformed UTF-8/JSON/OTLP records and record-local size/structure errors are retained while later valid records import. --strict stops on those errors. Output reports valid/invalid records, blanks, raw spans, raw logs, identical duplicates, conflicting segments and source-qualified notices. Single-document mode uses the bounded existing JSON artifact loader.

Hashes identify exact supplied file bytes and canonical source-span content, including content omitted from reports. They provide reproducibility against the supplied export, not source authenticity. Relative input paths stay under an explicit root; symlinks/junctions below it, remote files, archives and media reads are unsupported. Snapshot concurrently written artifacts before import.

Native files/receipts/inventory have stable bytes on repeated import. Conflicting output or trace mutation after receipt capture fails instead of overwriting. Native trace schema 1.1 and dependency/package versions remain unchanged. Older readers that ignore qualification metadata must not analyze these files.

Privacy

Input/output, unknown string attributes, log bodies, exception messages and reasoning are minimized before native parsing. Numeric/boolean and known structural attributes retain their semantics. Known credential fields, reasoning and token-ID fields stay omitted even with the Python-only capture_content=True option. Captured text needs review before sharing. Structural names/identifiers and attribute keys can still be private.

Unknown content is represented by a size/hash/omission marker. No payloads or decoder exception text enter diagnostics. Source resources, scopes, events, links, GenAI/OpenInference/MCP fields and external evaluations retain their supported noncontent semantics through the existing OTel parser.

Python and verification

from agentloop.interoperability.otlp_jsonl import OtlpOptions, import_otlp

result = import_otlp("harbor-traces.jsonl", options=OtlpOptions(
    system="harbor", identity={"job_id": "recorded-job-id"},
))
result.write("runs/imported")
print(result.inventory())
uv run --frozen python -m pytest tests/test_otlp_jsonl.py tests/test_otel_interop.py tests/test_harbor_atif.py tests/test_harbor_trials.py tests/test_interoperability_contracts.py -q -ra

Ordinary tests read frozen JSON only. Optional regeneration uses Python 3.12+ with the explicitly selected trusted converter source and recorded provenance:

python tests/fixtures/external/build_converter_fixtures.py --converter-source /path/to/reviewed/source --source-provenance /path/to/source-provenance.json

That command deliberately executes the selected upstream conversion modules. It is separate from ordinary package import, CLI use and tests, and requires no Harbor runtime, uploader, task verifier or vendor credentials.