Real Observability for Small Local LLMs with OpenTelemetry and COS Lite

Open source AI is often discussed as a model availability problem: can we run an open model locally? For real systems, that is only the first question. A small local LLM can answer successfully and still be unusable because the first token arrives too late, prompts burn too much CPU and token budget, responses are malformed, quality is poor, or gateway/model-selection errors hide behind green Linux and container metrics.

This talk presents a reproducible, CPU-first Ubuntu demo that turns those failures into measurable signals. The lab uses MicroK8s, Juju, COS Lite, Charmed Tempo, Ollama, a FastAPI OpenAI-compatible gateway, Canonical's Charmed OpenTelemetry Collector, Prometheus, Loki, Tempo, Grafana, and Alertmanager. The gateway emits OpenTelemetry GenAI spans, metrics, and trace-correlated logs while a scenario-driven load generator injects normal, slow, expensive, error, and low-quality traffic.

The talk focuses on the operational signals that matter for local and resource-constrained AI systems: time-to-first-token for streamed responses, token burn, per-request traces, alerting, and a simple quality signal. It also compares llama3.2:1b and qwen3.5:2b on the same CPU workload, showing why local model choice should be measured rather than guessed.

This is not a vendor dashboard tour. It is a practical FOSS operations story about using open telemetry patterns to decide whether a local AI workload is useful on the hardware available to the community. Attendees leave with a concrete mental model and a reproducible scaffold for observing small LLMs with familiar open source tools.

- Green infrastructure dashboards do not prove a local LLM is useful.

- Time-to-first-token matters more than total duration for streamed LLM UX.

- Token burn is an operational and cost signal, even for local models.

- OpenTelemetry GenAI telemetry can connect model behavior to traces, metrics, and logs.

- COS Lite, Prometheus, Loki, Tempo, Grafana, and Alertmanager are enough to build a useful LLM golden-signals view.

- Small model selection should be backed by workload telemetry, not only benchmarks or model size.