I Swear the Database Was Alive 5 Minutes Ago

While deploying PostgreSQL on Kubernetes is generally advertised as "highly available," reality tells us that there is much more to distributed failures. Node outages, split-brain conditions, corrupted WAL files, network partitions, increased storage latency, and cascade failovers can quickly lead to catastrophic cluster state from otherwise healthy systems.

In this talk, we will explore the subtle failure modes of PostgreSQL on Kubernetes by conducting chaos engineering experiments as well as infrastructure-level simulation tests. By leveraging the power of Kubernetes, Patroni, StatefulSets, persistent storage, and monitoring tools such as Grafana, Prometheus, Loki, and Tempo, we will deliberately cause disruptions to the PostgreSQL cluster in order to investigate replication lag, quorum failures, and failover storms.

Stateful services are becoming some of the most important and least understood elements within the cloud-native landscape. Whereas Kubernetes has grown up considerably to handle stateless workloads, running PostgreSQL in the face of failure is an ongoing problem that haunts platform engineers and site reliability engineers (SRE).

As companies migrate their highly available databases to Kubernetes, problems such as split-brain scenarios, replication delays, unreliable quorums, and WAL corruption, among others, have become commonplace but are generally only mentioned in relation to simplistic high-availability solutions. The presentation will address the operational challenges of running distributed databases in the face of failure using Chaos Engineering techniques, along with troubleshooting and recoverability analysis using Grafana and Prometheus monitoring tools.