Saved by Open Source: Surviving GitHub's Mood Swings With Tekton

August was a very rough month. On the 6th, Actions was degraded for over nine hours, with most workflow runs failing at the peak. On the 17th, a GitHub-wide outage took out most of another working day. In between, our team kept refreshing githubstatus.com like it was a cricket score.

We build disaster recovery for prod, databases and regions, then we ship every fix through one hosted CI provider and hope for the best.

And no, self-hosted runners don't save us. In most Actions incidents the runners are fine; sometimes it's the job scheduler that breaks. Jobs sit in "Queued" and nothing picks them up, hosted or self-hosted, because every runner waits for GitHub to hand it work. Your infrastructure is up. Your hotfix isn't going anywhere.

So we built a second road out that doesn't touch GitHub at all: Tekton, an open-source, Kubernetes-native pipeline engine that runs inside our own cluster as CRDs. When Actions goes down, we're shipping again in under 15 minutes.

We tested it against the outages GitHub has actually had:

1. Actions outage: The one that keeps recurring, scheduler being unable to schedule new jobs. Tekton schedules its own work in-cluster, so a queue stuck on GitHub's side doesn't touch it.

2. Webhooks outage: Rarer, but it happens. Without the webhook, nothing triggers the pipeline. So we added a GitHub App that polls for changes as the fallback trigger.

What we'll walk through

1. Anatomy of a GitHub Actions outage: which component fails, and why self-hosted runners still depend on it

2. The failover architecture, and the switch that moves releases onto Tekton

3. How we simulated each outage type, and the recorded drill

4. What it takes to make the second path real: the registry credentials and deploy secrets that only lived in GitHub

Everything in the failover path is FOSS: Tekton, Kubernetes, and the config we'll share.