Kafka on Kubernetes: A Production Migration Story from ZooKeeper to KRaft

· Apache Kafka deployments on Kubernetes, typically managed using Statefulset controller, have historically relied on Zookeeper for metadata management. With the transition to KRaft, Kafka's native metadata quorum, the Zookeeper dependency is being eliminated.

· Kafka 4.0, released in March 2025, marks the end of the Zookeeper era, making KRaft migration an immediate consideration for organizations running Kafka 3.x or older clusters.

· Apache Kafka Community has provided a standard multi step metadata migration procedure from Zookeeper to KRaft. Implementation of these steps presents number of challenges like applying config changes on Statefulset more than once with rolling restarts, monitoring of migration process, upgrade-rollback compatibilities etc.

· Drawing from our experience operating Kafka at Nokia, we built a Kubernetes Operator–based framework to orchestrate and automate ZooKeeper-to-KRaft migrations in production. We are exploring open souring a generalized version of this operator too.

· This talk explores the pain points involved in this migration process and how Operator based solution dealt with the same at production scale.

· Attendees get to know the practical pain points involved in this evolution of Kafka, especially for Kubernetes based deployments.

· Covers some of the strengths & weaknesses of the Kubernetes Operator framework and how it can be used in automating this complex migration within Kafka.

· Attendees will get to know the production grade challenges faced and the solutions to address the same. Examples; constraints in resource footprint increase during migration, monitoring of the migration procedure etc.