- LLM observability is not a dashboard of uptime. It is the ability to pull up one real request, see the exact prompt the model received, and explain why the answer was good or bad.
- Record the whole trace of a request, not just the final output: the resolved prompt with its injected context, the model and version, token counts, the latency split, every tool call, and the cost.
- OpenTelemetry now defines a shared vocabulary for this under the gen_ai namespace, so a LangChain span, a raw OpenAI call, and an agent step can all be read the same way.
- By default the standard captures no prompt or tool content, because those carry customer data. Turning content capture on is a privacy decision, not a checkbox.
- Page a human on four things only: an error rate spike, latency past your budget, a cost or token surge, and a drop in an automated quality score. Everything else is a dashboard, not an alarm.
A feature that summarizes support tickets worked all through testing and then, on a Tuesday, started returning summaries that missed the actual complaint. Nobody shipped a new model. The logs held thousands of prompts and thousands of responses. What they did not hold was the one thing that would answer the question, which change, on which request, made the quality drop. That gap is the whole subject of LLM observability, and closing it is mostly a decision about what you record before anything goes wrong.
The instinct is to log the input and the output and move on. That is enough to prove the API returned something. It is not enough to explain why the something was wrong, because the interesting failures in a language model feature happen between the input you wrote and the output you saw, inside the context that got assembled, the tools that got called, and the version that quietly changed underneath you.
How is observability different from monitoring?
Monitoring tells you the service is up. Observability tells you why a specific request behaved the way it did. The split is clean: monitoring watches known signals against thresholds, while observability lets you ask new questions of the system after the fact without shipping new code to answer them. For a normal web service the two blur together, because a 500 error is a 500 error. For a language model feature they pull apart hard, because the failure that matters is rarely an error at all. The call succeeds, the status is 200, the latency is fine, and the answer is wrong. No threshold catches that.
The reason is the material. A language model feature is non deterministic, sensitive to small prompt changes, and expensive per call, and its quality cannot be read off a status code. Those four properties are exactly why the observability tools built for ordinary services leave you blind here, and why a separate practice grew up around the trace of a single model request. If you have never priced what one of those requests costs to run, our breakdown of what a language model call actually costs in tokens is the companion to this piece, because cost is one of the signals you will be recording.
What should you record for one request?
Everything you would need to reconstruct the request without the original code in front of you. That is the test. The list below is what a single agent request is made of, and each row is a signal worth a span of its own.
OpenTelemetry has turned this from a matter of taste into a standard. Its GenAI semantic conventions define a shared set of attribute names under the gen_ai namespace, so instrumentation from different libraries records the same fields. The official OpenTelemetry write up on tracing inside the LLM call lists the core ones: gen_ai.request.model for the model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for the counts, gen_ai.response.finish_reasons for why generation stopped, and gen_ai.client.operation.duration as the latency histogram. The table below maps each signal to the question it answers and to what you lose when you skip it.
| Signal to record | Standard attribute | Question it answers | What breaks if you skip it |
|---|---|---|---|
| Resolved prompt with injected context | gen_ai.input.messages (opt in) | What did the model actually see? | You debug the template you wrote, not the prompt that ran |
| Model and version | gen_ai.request.model, gen_ai.response.model | Did the model change under me? | A silent provider update looks like your bug |
| Tokens in and out | gen_ai.usage.input_tokens, output_tokens | Why did this call cost what it cost? | Cost spikes have no explanation |
| Latency, split by phase | gen_ai.client.operation.duration | Was it slow to start or slow overall? | You cannot tell a slow model from a slow tool |
| Tool calls, arguments and results | gen_ai.tool.name, gen_ai.tool.call.id | Which step made the wrong decision? | An agent failure is a black box |
| Finish reason | gen_ai.response.finish_reasons | Did it stop, or get cut off? | Truncation reads as a bad answer |
| Sampling parameters | gen_ai.request.temperature, top_p, max_tokens | Was the output shaped by a setting? | You chase randomness that a config caused |
The two rows people forget are model version and finish reason. A provider can update the model behind a stable name, and if you did not record gen_ai.response.model you will spend a day blaming your own prompt for a change you did not make. A finish reason of length rather than stop means the answer was cut off by a token limit, which looks identical to a bad answer if you only read the text. Recording both is close to free and saves the two worst debugging days.
One more field earns its place: a conversation id. A field level guide to the GenAI conventions uses gen_ai.conversation.id to group the turns of a multi turn exchange, and without it a ten step agent run scatters into ten unrelated spans. Grouping is what turns a pile of logs into a trace you can read top to bottom, and reading it top to bottom is the entire point.
Which kind of tool do you actually need?
Three categories exist and they are genuinely different, so buying the wrong one leaves a gap you will not notice until an incident. PostHog's survey of open source observability tools draws the first line between integrated platforms that fold model tracing into broader product analytics and specialized tools built around the trace and the evaluation loop. There is a third shape the two miss, the gateway that sits in the request path and gets cost and caching for free while seeing nothing inside your application logic.
| Tool category | What it is genuinely for | What it sees | What it misses |
|---|---|---|---|
| General purpose APM | Tokens, latency and errors alongside the rest of your service | The call as one span in a wider trace | Prompt content and evaluation quality |
| AI native tracing | The full trace, prompt versioning and eval scoring | Every step, tool call and quality metric | Nothing model side, but it is another system to run |
| Gateway or proxy | Routing, cost attribution and caching in the request path | Cost and traffic across providers | Your application logic and why an answer was wrong |
The distinction that trips people is the last column. A gateway that sits in front of every provider gives you cost and caching almost for nothing, because it is already in the path, but it cannot tell you why an answer was wrong, because it never saw the retrieval step or the tool result that poisoned it. That is a different tool from the tracing platform that watches inside your code. Many teams end up running both, and the reason they overlap without being the same is worth understanding before you buy either. We go deeper on where that line sits in the piece on what an LLM gateway centralises, and the short version is that a gateway owns the request path while an observability tool owns the trace.
Whichever you pick, prefer instrumentation that speaks the OpenTelemetry gen_ai conventions. PostHog notes that OpenTelemetry compatibility is what lets model traces sit inside the infrastructure you already run, rather than becoming a second, disconnected pane of glass. Standard attributes also mean you are not locked to one vendor, since the same spans export anywhere. If you are still deciding which model to send those requests to in the first place, our method for telling whether a prompt change actually helped pairs with observability, because evaluation is the quality signal your traces should carry.
You rarely write this instrumentation by hand. Most model SDKs and agent frameworks now ship auto instrumentation that emits gen_ai spans for you, so the work is less about writing spans and more about deciding what to keep, what to sample, and what to redact. That is why the standard matters more than any single tool. When the attribute names are shared, you can start with an off the shelf library today, swap the backend later, and keep the traces you already collected, instead of re instrumenting the whole application every time you change vendors. The decision that costs you is not which library to bolt on, it is whether you resolved, in advance, to record the full request rather than just its two ends.
What about the customer data in your prompts?
This is the part vendor pages skip. Full prompt capture is the most useful thing you can record and the most dangerous, because a resolved prompt often contains exactly the customer data your privacy policy promised to protect, and turning on content capture ships that data to your observability vendor. The people who wrote the standard understood this. By default, the OpenTelemetry GenAI conventions capture no prompt content and no tool arguments, precisely because they can hold sensitive data, and enabling content capture is an explicit opt in.
So the policy is not all or nothing. Record the structural signals always, since token counts and latencies carry no personal data. Capture content selectively, behind a redaction step that strips personal fields before the span leaves your process, and sample it rather than keeping every request forever. A common shape is to capture full content on a small percentage of traffic plus every error, which gives you enough real examples to debug without turning your trace store into a second copy of your customer database. Deciding what counts as sensitive here is the same inventory work that governance demands, and our guide to the first ninety days of AI governance covers how to make that list before an auditor asks for it.
Which alerts are worth paging on?
Almost none of them. The failure mode of observability is a wall of alerts nobody reads, and a language model feature generates plenty of noise if you let every wobble trip a pager. Four signals deserve to wake someone, and the rest belong on a dashboard you check, not an alarm that finds you.
The first is an error rate spike, meaning failed calls, rate limits, and timeouts crossing a rate you set, because that is your feature actually breaking. The second is latency past your budget, and here the split matters, since a rise in time to first token points at the provider while a rise in total time with a normal first token points at your own tool calls. The third is a cost or token surge, because a runaway retry loop or a prompt that quietly grew can multiply your bill overnight, and token count is the leading indicator that reaches you before the invoice does. The fourth is a drop in an automated quality score, which is the only one of the four that catches the Tuesday failure at the top of this piece, and it requires that you are running evaluations continuously rather than only before release.
Everything else, the individual slow request, the single odd answer, the one timeout, is a data point you investigate through the trace, not an interruption. The discipline is to keep the pager pointed at rates and trends, and to keep the trace rich enough that when a page does fire you can open one request and read the whole story. Getting to that discipline is a maturity question as much as a tooling one, and it sits inside the same ladder we lay out in the piece on what to build first as your ML operations mature.
What does reading one trace look like?
Go back to the Tuesday failure and assume you had recorded everything above. You open the quality dashboard, see the summary score dropped on Tuesday afternoon, and click into a low scoring request. The trace shows the resolved prompt, and there it is: the injected context now includes a new ticket field that a colleague added on Tuesday, and it pushed the real complaint past the point where the model stops paying attention. The finish reason is stop, so nothing was truncated. The model version is unchanged, so the provider is innocent. The tool call that fetched the ticket returned the new field at the top. You found the cause in one trace, and it was a context assembly change, not a model change, which is the most common real answer and the one that logs of inputs and outputs alone can never give you.
That is the shape of every good observability session. You move from an aggregate signal to a single request, read the request top to bottom, and the cause is usually a step you would not have thought to log if you had not decided in advance to log all of them. The cost of that decision is storage, which is the honest tradeoff nobody mentions.
How much should you keep, and for how long?
Not all of it, and not forever. Structural signals like token counts, latencies, and finish reasons are small and cheap, so keep them at full volume. The expensive part is content, the prompts and tool results, which can dwarf everything else in storage. A workable retention policy keeps structural spans for weeks and full content for a much shorter window plus a permanent sample of errors, so you can reconstruct a recent request in full while old traces decay to their cheap skeleton. Set that policy on day one, because a trace store that grows without bound becomes its own cost problem, and an observability bill that rivals the model bill is a failure of exactly the discipline this practice is supposed to teach.
If you are wiring an agent that calls tools, the trace becomes even more valuable, because the interesting failures move into the tool steps. That is also where a standard protocol for tool calls pays off, and you can see how those calls are structured in our walkthrough of building a tool server with the Model Context Protocol. Record the tool name, the arguments, and the result on every step, and an agent that misbehaved stops being a mystery. The reconstructed request is the deliverable. Build the trace so that any request, months later, can be opened and understood without the code, and you have the thing a dashboard can never give you.