Blackbox: eBPF Post-Mortems for Linux Scheduling Stalls

Debugging a Linux server that hangs or gets OOM-killed can be difficult, especially when the issue happens in production and cannot be reproduced. By the time someone investigates, the process may already be dead, and the available logs might not explain what happened.

I built Blackbox to address this problem. It is an open-source Linux post-mortem recorder written in Rust using eBPF and the aya framework. It records context switches across the machine, maintains a bounded rolling window of recent events, and saves the trace when it detects memory pressure or an OOM kill. The output is a self-describing JSON trace that can be read as a text timeline or opened in Perfetto to inspect what happened around the failure.

In this talk, I will explain how scheduler traces can help investigate stalls, including how to interpret context switches, task states, and CPU scheduling behaviour. I will also walk through the implementation, covering the Rust and eBPF components, CO-RE, ring buffers, and PSI and OOM triggers.

I will share the performance measurements from the project, including the initial overhead of 26–63% on switch-heavy workloads and how an in-kernel PID filter reduced it to 1.5–4.7%. I will also cover memory usage, the limits of the recorder, and the security decisions behind the daemon and its control socket.

The talk will include a live demo where I induce memory pressure, let Blackbox capture the event, and inspect the resulting trace to understand what was running, what was waiting, and which process was killed.

Blackbox is fully open source and uses Rust, aya, and Linux tracepoints without requiring a custom kernel.

1. Understanding Linux scheduling stalls: How to read context-switch records, task states, PIDs, and CPU information to understand what was happening when a system stalled.

2. Capturing failures for post-mortem analysis: How to maintain a bounded rolling window of scheduler events and trigger a dump when PSI pressure or an OOM kill is detected.

3. Building eBPF tools with Rust: How Blackbox uses aya, CO-RE, ring buffers, and Linux interfaces such as /proc/pressure and /proc/vmstat without requiring a custom kernel.

4. Managing overhead and memory usage: What the benchmarks show, how PID filtering reduces collection overhead, and why the recorder needs clear limits and reporting for missing events.

5. Security and operational considerations: How the daemon is designed to run with limited exposure, without network access, and with access to sensitive control operations restricted using SO_PEERCRED.