Scaling Real-Time Upserts in Apache Pinot: Reliability and Efficiency at Any Scale

Modern data-driven organizations require analytics platforms that can not only ingest high-velocity data streams but also support rapid updates and corrections on mutable datasets—a challenge that traditional OLAP engines struggle to meet. In this talk, we’ll explore how Apache Pinot delivers industry-leading throughput for upserts without compromising on efficiency or reliability.

We will explain how Pinot enables both high-throughput ingestion and low-latency queries without sacrificing correctness. We’ll explore Pinot’s metadata design, segment-level bitmaps, and partitioning strategies that keep ingestion and query overhead minimal. Real-world scaling bottlenecks—such as memory constraints and server restarts—will be addressed, along with solutions introduced by the community and StarTree Cloud: off-heap metadata management with RocksDB, minion-based metadata prebuilding, and efficient compaction strategies.

• Learn the internal architecture that makes upserts in Pinot efficient and reliable

• Discover practical strategies for scaling upserts to billions of keys per server

• Features ensuring query accuracy and performance under heavy, concurrent workloads.

• See how StarTree Cloud extends Pinot’s upsert capabilities for massive scale