Why LLM Observability is the New Frontier in AI Safety

Large Language Models (LLMs) are no longer just research toys, they're in production, making decisions, influencing users, and shaping outcomes at scale. But here's the hard truth: you can't trust what you can't see. LLMs can fail silently, hallucinate with confidence, and degrade without warning. "Trust but verify" isn't optional it's the new baseline for responsible AI.

This session cuts to the core of the issue: traditional observability wasn't built for non-deterministic, black-box systems like LLMs. We'll explore why LLM observability is fundamentally different and how we can build the tools to meet that challenge.

Open source is leading the charge toward AI transparency. If you're deploying LLMs into Kubernetes-native stacks or building safety-critical AI services, this talk will give you a roadmap to make your models observable, measurable, and verifiable just like the rest of your infrastructure.

A key part of the solution lies in OpenTelemetry, the open-source standard for instrumenting distributed systems. By extending its tracing and metrics pipelines to capture LLM-specific signals like prompt/response payloads, latency, cost, token usage, and model confidence, we can build visibility into how these models behave across real-world workflows.

We'll explore:

• How OpenTelemetry enables structured tracing of LLM chains, across micro-services and vector stores.

• What to log and trace for LLMs prompt/response deltas, temperature drift, user feedback, error modes, and more.

• How to pair OpenTelemetry with Prometheus, Grafana, and CNCF-native tools to create actionable observability dashboards.

• Real-world patterns for monitoring prompt injections, data leakage, and performance regressions in production LLM apps.