Overengineering the BRC: Building a Distributed Benchmarking Platform

The original plan was simple: host our own version of the Billion Rows Challenge, let people submit Python implementations, and automatically benchmark them. We thought the infrastructure would take around five days. It took five weeks.

Our first attempt was GitHub Actions, which quickly ran into storage limits, concurrency limits, expensive rebuilds, security problems, and very little control over how arbitrary participant code was executed. So we started replacing pieces ourselves.

The final system listened for repository commits through a GitHub App, pushed jobs through RabbitMQ, built and executed submissions inside containers, and used a Rust benchmarking tool around the participant's Python code. We then tried to scale workers using Terraform, Kubernetes and KEDA based on queue depth.

That worked beautifully on the architecture diagram. Reality included 17-minute container builds, debugging autoscaling that wasn't actually broken, Azure vCPU quotas, and eventually distributing workers across multiple accounts because the infrastructure we designed couldn't be deployed the way we expected.

This talk is about that journey: the architecture, the bad assumptions, the tools that worked, the parts we threw away, and what we learned about queues, reproducible environments, benchmarking, untrusted workloads and designing systems around real-world constraints rather than ideal ones.

• When a queue and worker architecture makes more sense than trying to stretch CI/CD into a compute platform.

• How we used GitHub webhooks, RabbitMQ and isolated workers to turn repository commits into benchmark jobs.

• What Docker solved for reproducibility, and what it did not solve when running and benchmarking untrusted code.

• Why autoscaling architecture can be correct while still failing because of cloud quotas and deployment constraints.

• How moving test generation from Python to Rust reduced generation time from roughly 23.5 minutes to around 25 seconds.

• What we would simplify if we were designing the system again, and where the original overengineering actually turned out to be useful.