Beyond Alert-as-Code: Building a GitOps Governance Platform for Production Alerting

In my work as an SRE, we operated hundreds of business-critical production alerts supporting customer journeys, platform services, operational monitoring, and revenue-impacting systems across multiple teams.

These alerts were implemented using Elasticsearch Watchers, a framework for defining scheduled alerting logic directly within Elasticsearch. Initially, the problem looked simple: move alert definitions into Git, validate them, and deploy them through CI/CD. But once alerting became code, we discovered a deeper set of problems.

• Who owns an alert?

• Who reviews the logic before it reaches production?

• How do we prevent duplicate, noisy, or unsafe alerts?

• How do we know whether an alert is still useful?

• How do we report on alert quality, alert fatigue, and operational impact?

Solution:

To solve this, we started as a GitOps repository for managing Elasticsearch Watchers as code, but evolved into a governance and reporting layer for production alerting. It supports watcher creation, validation, deployment, deletion, metadata enforcement, inline policy checks, review workflows, reporting, and a UI for building and managing watchers safely.

Architecture-wise, our solution brings together:

• GitLab as the source of truth for alert definitions

• CI/CD pipelines for validation, deployment, and deletion

• Elasticsearch Watchers as the alert execution engine

• Kibana and Elasticsearch history indices for alert firing data

• Policy checks for naming, metadata, ownership, severity, and environment safety

• A self-service UI for watcher creation and governance

• Reporting jobs to analyse watcher firing history, noisy alerts, stale alerts, and coverage gaps

The business requirement was not just “deploy alerts faster.” The real requirement was to make alerting auditable, reviewable, measurable, and safe at production scale.

This talk will walk through how we designed, the architecture, decisions we made, the failure modes we encountered, and what we learned while converting alerting from hidden monitoring configuration into governed software artifacts.

This is a production story about platform engineering, SRE governance, and building decision intelligence on top of open-source operational systems.

1. Why Alert-as-Code is only the first step, not the final architecture.

2. How GitOps can make alerting reviewable, auditable, and safer for production teams.

3. How to design a governance layer around alerts using metadata, ownership, validation, and policy checks.

4. How alert firing history can be converted into operational reporting and decision support.

5. How a self-service UI can reduce operational dependency while still enforcing platform standards.

6. Lessons learned from running alert governance across production systems, including noisy alerts, duplicate logic, stale watchers, and unsafe changes.