From Claims to Evidence: Testing Reliability in FOSS

FOSS projects earn trust in public. But many important project claims, like highly available, reliable, or production ready, often live as statements in documentation instead of experiments that users and contributors can inspect.

In this lightning talk, I will share a lesson from my Linux Foundation mentorship: how we turned one reliability claim in an open source PostgreSQL project into a repeatable test. The interesting part was not the infrastructure. It was deciding what promise the project was making, what evidence would convince a skeptical user, and how to package the result so future contributors could rerun it.

I will walk through the shape of the experiment: choose one claim, define the failure, check the system's behavior, save the artifacts, and make the result reviewable. I will also share a small upstream contribution that came out of the work, where a hidden assumption in a testing tool made the experiment less precise than it looked.

The goal is for maintainers and contributors to leave with a practical question they can apply to their own projects: what does our project ask users to trust, and how can we make that trust reproducible?

1. Treat project claims as promises that users should be able to verify.
2. Start with one important promise and design a small, repeatable check for it.
3. Publish enough context, results, and artifacts for others to inspect the evidence.
4. Use testing work as a contribution path, even when you are not a core maintainer.