When Custom Controllers Go Rogue: Postmortems from Building Kubernetes Operators in Go

We often treat Kubernetes operators as magic black boxes that seamlessly manage complex workloads. But what happens when that magic turns into an infinite reconcile loop that brings down your entire control plane? 
This 10-minute talk dives straight into the trenches of custom controller development in Go. I will share real postmortems involving runaway memory leaks, stale reads from the API server cache, and unintended race conditions. We will cover how to correctly implement rate-limiting in workqueues, manage complex state transitions safely, and instrument your operators with Prometheus so you catch failures before your users do. This is a battle-tested guide for platform engineers looking to write, debug, and scale resilient Kubernetes extensions in production.

• Common anti-patterns in Go-based Kubernetes controllers that lead to API server exhaustion.

• How to debug complex state drift and cache invalidation issues in custom operators.

• Essential Prometheus metrics and tracing strategies for deep operator observability.

• Best practices for rate-limiting and exponential backoffs in Kubebuilder workqueues.