LLM observability: seeing what the model did in every request
Record each LLM request as a trace: prompt, response, tools, tokens, latency and cost. Learn what to store and how to use it to debug and measure quality.
What is a trace in an AI system?
A trace is the full story of one request. If a user asks about a cancellation, the trace shows that the system received the message, called the model with a certain prompt, the model asked to look up a booking, the tool returned a result and the model answered. Each step has its own duration, inputs and outputs, and the steps are linked to each other.
The Langfuse documentation describes tracing as structured logs of every request that record the flow of the request through the system and preserve the causal relationships between operations. In its interface those operations appear nested, with timing, inputs, outputs and metadata. LangSmith, from LangChain, defines a trace along similar lines: the record of what your agents did in production, used to debug failures, monitor quality and build evaluation datasets.
What is worth recording?
A useful trace stores, at minimum:
- The exact prompt sent, with the prompt version that was active.
- The model’s response and why it stopped (complete answer, token limit, tool call).
- Tool calls and document retrieval, with their inputs and results.
- Input and output tokens, to measure cost.
- Latency per step, to know where time goes.
- The model identifier and the application version, to compare deployments.
On cost, Langfuse explains that it uses the cost you send with the model response and, if there is none, infers it by multiplying the token usage you report by the model’s price in a table of model definitions. That is why the provider or your code must report the tokens: it does not estimate them from the text. To see why that number matters, read cost per token.
What standard exists for naming that data?
OpenTelemetry, the open telemetry standard, maintains semantic conventions for generative AI. They now live in their own repository (semantic-conventions-genai), and the index marks the set as Development, meaning not stable. They cover inference, embeddings, tool execution, retrieval and agents, plus specific conventions for Anthropic, OpenAI, AWS Bedrock and Azure AI Inference. A few attribute names from the inference page:
| What you record | OpenTelemetry attribute | Level |
|---|---|---|
| Operation type | gen_ai.operation.name | Required |
| Requested model | gen_ai.request.model | Conditional |
| Input tokens | gen_ai.usage.input_tokens | Recommended |
| Output tokens | gen_ai.usage.output_tokens | Recommended |
| Input and output messages | gen_ai.input.messages, gen_ai.output.messages | Opt-in |
Two practical consequences. First: because the standard is not yet stable, names can change, so check the version of your instrumentation library before building dashboards on a specific attribute. Second: message content is opt-in on purpose. The specification warns that those fields are likely to contain sensitive information, including personal data.
Practical advice: use an open format when you can, so you are not tied to a single tool.
Why does it matter to a designer?
Because observability is what lets you design the response to failures. If you discover that a tool runs a slow query and that delays many answers, you can design a loading state specific to that step. If you see the model retry an action several times, you can change the interface to ask for confirmation.
It also changes the conversation with engineering. Instead of “the assistant feels off”, the team can point to a concrete trace, its slow step and its response.
How do you use it for quality?
Langfuse proposes three uses, which work as a guide:
- Debug. According to its documentation, debugging without observability is closer to guesswork. With traces, you see what happened in each request.
- Monitor quality. Attach scores to traces and raise alerts when a metric crosses a threshold you define.
- Evaluate. Production traces are real data to run evals on and to improve the system continuously.
In their evals guide, Hamel Husain and Shreya Shankar recommend starting with error analysis on real outputs, reviewed by hand (they mention 20 to 50), before reaching for generic metrics. Observability is the source of those traces.
When should you NOT do it?
- When you store full content without deciding what happens to it. Prompts and responses can include names, phone numbers or health data. See privacy in AI products.
- When nobody reviews the traces. A dashboard nobody looks at does not improve the product.
- When everything is logged without filtering. Noise hides the failures that matter.
Example
Illustrative example: an assistant in a travel app gets complaints that it “sometimes takes too long”. The team turns on traces with latency per step and the prompt version. Within two days they see that a share of requests exceeds 14 seconds, and that almost all of that time is a call to the flight availability tool, not the model. With that evidence, the team changes the interface: it shows a loading state naming the query in progress, and asks engineering to cache availability for five minutes. Two weeks later the traces show whether the share of slow requests dropped.
The same traces later feed the error analysis described in evals as UX research. For loading states on screen, read loading states in AI interfaces. To measure how the feature works as a whole, see quality metrics for an AI feature. And if the system acts on its own, the agent loop shows which steps are worth surfacing in the trace.
Common mistakes
- Recording only the final text and not the intermediate steps.
- Storing full prompts and responses without deciding their retention or who can see them.
- Not recording the prompt version. Without it, you cannot compare before and after a change.
- Building dashboards without alerts or an owner.
- Confusing the model’s latency with total latency. Often the bottleneck is a tool.
- Depending on experimental attribute names without pinning the instrumentation library version.