Events, traces, metrics, and correlation
Use structured events, span relationships and aggregate measurements to reconstruct and measure an agent run.
- Distinguish logs, events, traces, spans and metrics.
- Correlate attempts, operations and runs across services.
- Interpret latency and error measurements without equating transport success with task quality.
Why this matters
Section titled “Why this matters”Lee says the assistant took too long to answer. A final response cannot tell us whether retrieval was slow, the model waited on a tool, an attempt failed or the workflow paused for human input. We need records from the components that performed the work.
Observability is the ability to understand a system’s behavior from its outputs and records. Telemetry is the measured or recorded data it emits. Begin with questions: Where did time go? Which action completed? What was retried? Was the system waiting for human input?
How it works
Section titled “How it works”Give records different jobs
Section titled “Give records different jobs”An event describes something that happened, such as approval_received. A log is a timestamped record, often carrying an event and its fields. A trace follows an end-to-end operation across related steps. A span represents one timed operation within a trace. An attribute is a named value attached to a record, such as a tool name or attempt number.
OpenTelemetry spans carry identifiers, parent relationships, time bounds and status; they can also contain events and links to other spans. Parent-child relationships describe operations within larger operations. Links can connect asynchronous work without forcing everything into one nested lifetime. OpenTelemetry traces.
For our shop, a run span contains retrieval, a model request and tool attempts. A handoff records transfer to another worker, whose summary operation is related to that transfer. An approval request is a point event; time spent waiting for the decision can be measured separately. These are observable runtime operations, not a transcript of hidden model reasoning.
Figure 1. Start at the run, then inspect its child operations. The two tool attempts belong to one logical lookup. A shared trace ID joins the records; separate span IDs identify individual operations.
Correlate the right identities
Section titled “Correlate the right identities”A correlation identifier connects related records. Different identifiers answer different questions:
| Identifier | Question it answers in our design |
|---|---|
| Session ID | Which continuing conversation contains this run? |
| Run ID | Which attempt to fulfill a task is this? |
| Trace ID | Which instrumented operation contains this span? |
| Span ID and parent ID | Which timed step is this, and what contains it? |
| Model call ID | Which model-requested call receives this output? |
| Operation ID and attempt number | Which retries belong to the same logical action? |
Context propagation transports trace context between components, allowing a receiving service to relate its work to the caller’s span. It requires cooperation at the boundary. A trace header is correlation data, not proof that the caller is authorized. Context propagation.
Use causal relationships and sequence numbers alongside timestamps. Machines can have different clock settings, so sorting logs by time alone may misrepresent distributed ordering. The mock uses one synthetic clock to make its timing checks deterministic.
Summarize runs with metrics
Section titled “Summarize runs with metrics”A metric measures behavior numerically across observations. A counter accumulates occurrences, such as completed tool attempts. A gauge reports a current value, such as waiting tasks. A histogram groups measured values, such as durations, into a distribution. OpenTelemetry defines these instrument types and aggregation mechanisms. Metrics.
Our mock has one logical lookup, two attempts and one attempt error. The complete run succeeds. “Tool error rate” therefore needs a defined denominator: attempts and logical operations are different populations. User-visible task success is another measurement, requiring evaluation against the goal.
Latency measures elapsed time. Distinguish end-to-end response time from retrieval duration, tool duration and human wait time. Parallel span durations can overlap, so adding every child duration is not a general way to calculate total latency.
Cardinality is the number of distinct attribute combinations. Aggregate metrics by bounded categories such as tool name and outcome. A separate metric series for every run ID or full prompt can create an unbounded set. Preserve individual identifiers in records intended for per-run inspection.
Keep the application contract stable
Section titled “Keep the application contract stable”OpenTelemetry’s GenAI agent conventions describe operations such as agent invocation, workflows and tool execution. They are currently marked Development, so implementations should pin a version and map to a stable application schema. GenAI agent spans.
Record measured model usage when available, with units and provenance. Missing token usage is not zero usage. A cost estimate also depends on the applicable pricing and caching contract, not just a span duration. This guide does not embed a pricing assumption.
Minimal implementation
Section titled “Minimal implementation”The OpenTelemetry-shaped offline example creates a trace with a retrieval step, model step, two tool attempts and a handoff. Assertions check identifiers, parent links, time ranges and metric denominators. It prints JSON to the terminal and exports nothing.
Expand minimal Python implementation
tools = [s for s in spans if s["name"] == "lookup_stock"]counts = Counter(s["status"] for s in tools)operations = {s["attributes"]["project.operation_id"] for s in tools}The simplified shape and project.* attributes are our teaching schema, not the exact OTLP wire format. OTLP is OpenTelemetry’s protocol for transporting telemetry.
Framework implementations
Section titled “Framework implementations”The OpenAI Agents SDK supplies tracing for model generations, tool calls and handoffs, with configuration and custom processors. Its identifiers and span data differ from OpenTelemetry’s. Map the shared concepts and preserve required relationships when integrating the systems rather than renaming an SDK JSON object “OTLP.” SDK tracing.
Limitations and trade-offs
Section titled “Limitations and trade-offs”Sampling retains only selected telemetry, reducing volume while limiting what can be reconstructed. Missing spans can also reflect instrumentation gaps or export failures. A quiet dashboard does not establish that no operation happened.
Record known outcomes at the component that observes them. A model’s final statement that a reservation succeeded is different from the order service’s receipt. Traces help locate a discrepancy, while evaluation determines whether the resulting answer fulfilled the task.
Security preview
Section titled “Security preview”Telemetry can contain sensitive data. Pass 2 examines that exposure and detection needs in human interfaces and observability security. Useful reconstruction does not require recording every raw prompt.
Key takeaways
Section titled “Key takeaways”- Events describe changes, spans time operations, traces connect them and metrics summarize measurements.
- Preserve run, span, call and operation identifiers for their different purposes.
- Define denominators and units before interpreting error rates or latency.
- Observable execution success and task quality require different evidence.
Check your understanding
Can a successful run contain a failed tool attempt? Yes. A later attempt may succeed; attempt errors and operation outcomes use different denominators.
References
Section titled “References”- OpenTelemetry. Traces, metrics, and context propagation.
- OpenTelemetry. GenAI agent and framework spans. Development-stage conventions, verified for this run.
- OpenAI. Agents SDK tracing. Living documentation, verified for this run.
The next chapter is an optional lineage and telemetry deep dive. Readers staying on the main path can continue to evaluation and benchmarks.
