7+ million Postgres tables

The talk explores how we use a single node Postgres to have more than 7 million ephemeral tables as a caching strategy for serving reports from a ~trillion-row database to millions of users on a daily basis at Zerodha. This is enabled by DungBeetle, a distributed SQL querying and caching system that we developed. It sits in front of large SQL databases to distribute and queue user requests for reports that involve running complex SQL queries that typically end up choking large databases. The talk explores the hacks and decisions that went into what originally seemed like a ridiculous idea, but turned out to be high performing, cost-effective, and practically zero maintenance solution to a hard problem.