Databases in Ethereum Nodes

General-purpose databases handle the overwhelming majority of workloads. Pick a flavor — relational, NoSQL, column-store, in-memory — tune its config, and it'll fit an enormous range of problems.

But in some fields, we need to go down to the fundamentals and build a custom database from the ground up. In bioinformatics, the need is often to store petabytes of data and/or have a database that can efficiently do string operations on structures like suffix trees or the Burrows–Wheeler Transform (FM-index). Blockchains are another such field. They work under different constraints — unique access patterns, authenticated state for a trustless environment, distribution requirements and different artifacts to store.

The Ethereum Foundation coordinates with a cohort of client teams that drive protocol research and development. Pretty much every client has come up with custom storage solutions. Erigon is one of these client teams; it specializes in archive nodes — the biggest node type by storage size, relied on by node/RPC providers (Infura, Alchemy, QuickNode) and by indexers and analytics platforms (The Graph, Dune, Etherscan).

In this session, I will talk about ErigonDB, the custom database devised by Erigon. It's a multi-tiered database with an mmap'd B-tree (MDBX) at L0 and custom snapshots at the lower levels. It provides transactional semantics, and the immutable snapshots are distributed via torrent to other nodes. We'll explore the data model, succinct data structures, custom file formats and algorithms that power ErigonDB — and of course the constraints and motivations that drove those decisions.

• access patterns in blockchains in general and ethereum in particular

• kinds of data that need to be stored in an ethereum node and the distribution requirements

• the architecture of the custom database; composing ACID properties

• different algorithms and data structures used in database - recsplit, elias-fano, existence filters, good old btree and so on.