From “is it up?” to alerts in minutes: MCP on Prometheus + Grafana

Title : From “is it up?” to alerts in minutes: MCP on Prometheus + Grafana

Session description

- This talk drives the Prometheus → Grafana → alert workflow through MCP tool calls instead of Grafana clicks, with a practical boundary for when MCP reduces toil vs when GitOps/IaC remains better.

- MCP, or Model Context Protocol, is an open protocol, originated by Anthropic and now multi-vendor, that lets clients such as Cursor and Claude Desktop call tools and pull data from sources in a uniform way. The demo uses Prometheus, Grafana OSS, and the Apache-2.0 open-source `grafana/mcp-grafana` server.

- Live demo, with a recorded fallback for connectivity: on a 5-target lab Prometheus fleet modelled on production scrape-health patterns, I will run PromQL on `up`, build an availability dashboard, and create Grafana-managed alert rules in under 5 minutes.

- This is from running Prometheus and Grafana day-to-day in an SRE setting; the demo lab is modelled on those scrape patterns.

- The fleet includes node, fluentd, pushgateway, app metrics, and prometheus jobs, which makes the demo realistic enough to show both per-target and fleet-level failure handling.

- What broke: Grafana was unavailable, the MCP server was not registered, early `up == 0` alerts were noisy, line charts were misleading for binary data, and dashboard refresh settings created unnecessary load.

- Operational lessons: `up == 0` means “scrape failed”, not always “server dead”; alert design needs per-target and fleet-level rules, pending windows, no-data behavior, and clear severity.

- Attendees will leave with a repeatable query → dashboard → alert workflow and security basics around Grafana tokens and least privilege.

- I will publish the MCP tool config, dashboard JSON, and alert-rule YAML in a GitHub repo before the talk so attendees can inspect and reuse the workflow.

Audience takeaways

• How up relates to scrape health and what it does not prove

• A repeatable loop: query → dashboard → alert via MCP tools

• Practical alert design: per-target vs fleet, pending windows, no-data behavior

• Security basics: Grafana tokens, least privilege, MCP has no built-in auth — you implement it

• When MCP helps (toil reduction) vs when traditional GitOps/IaC is still the right answer