Long live outbox : Outbox is dead

TLDR

• How we utilized a lesser known postgres feature/hack to inject custom data into the database WAL stream to remove the outbox table completely. Reduced tail latencies and make downstream events durable in distributed systems en route.

Background

• Roots from learnings of production challenges of dealing with with realtime payments, at #6th largest UPI payments org by volume

Context

• Outbox pattern is a well known pattern used in distributed systems to power side-effects of actions. These effects need to be strongly tied to actions, else users face broken processes. Need durability + reliaiblity + speed (fast rewards/cashbacks!)

  • Actions (SignUp, Transaction, …)

  • Effects: (Allow UPI Id, Bank Account Fetch, Cashback, ..)

  • Some examples of our usecases

    • Durability : services will go down, still need to give cashback

    • Reliability: users need cashbacks after the txn, always.

    • Speed: no one wants to wait 5 seconds to get a cashback

The Old Way

• Write data in a new database table wrapped in an atomic block → double writes and updates, contention on database disk and cpu resources.

• Run a polling based consumer to read→ dedicated index needed to poll, scan, read record. Send, handle retires, backoffs, polling delays, expensive housekeeping.

• Long Maintenance Windows: MVCC means updates are inserts, deletes are not actually deletes, but tagging → this cleanup needed everyday via vacuum jobs.

The Hack?

• To keep a standby in sync, the database writes every change to WAL log, ships those bytes to replicas in order, and tracks how far each replica has caught up.

• Postgres’ replication can already carry application’s events

• We ride the pg_logical_emit_message  to write a blob into the WAL atomically with the surrounding transaction, and consume it from WAL stream.

• We use this in production, will share benchmark results, consumer guarentees, gotchas, when to use and when not to etc.

1. Benchmark Results of the new method v/s classical method

2. Production risks and learnings of using this on millions of events per hour scale on prod.

3. How we removed an entire class of infra (outbox workers, retries, cleanup) from our system and The hidden cost of "industry standard" patterns → and how to avoid them