Lightning Talk Beginner MIT First Talk

Bad Internet, Good Software

Review Pending

Meet Mehta

Session Description

Our college internet was terrible. Not slow in a predictable way, but inconsistently slow, the worst kind. We tried everything we could find: aria2, wget retry flags, browser extensions, GUI download managers. Some were powerful but clunky, some abandoned, and many simply did not fit how we actually worked, in terminals, over SSH, across laptops and servers.
So we did what CS students usually do when the right tool does not exist: we built our own. At first, this was not a serious open-source project. We mostly wanted to understand how downloading works under the hood and maybe end up with a decent resume project.
That small experiment slowly became Surge, a terminal-first, headless-capable download manager written in Go. Along the way, it grew into a project that benchmarked around 1.4× faster than aria2 and unexpectedly crossed 3,000+ GitHub stars after a Reddit post we thought might get a few dozen reactions.
This talk tells the full story of how that happened, beginning with the fundamentals of modern downloading: HTTP range requests, CDN throttling, load balancing, and why splitting files into chunks can dramatically improve speed and reliability.
The real turning point came when we started benchmarking obsessively. We built an internal harness to track throughput, retries, worker stalls, chunk completion, and disk latency, and the results kept surprising us. Identical workers hitting the same URL often behaved completely differently.
Those benchmark graphs pushed us into research on concurrency, scheduling, and work-stealing systems. We experimented relentlessly, measured everything, discarded ideas that failed, and refined the ones that consistently worked. Features like dynamic chunk stealing, slow-worker replacement, and duplicate tail workers emerged directly from those experiments.
Beyond performance, this talk is also about open source. Maintaining Surge meant reviewing pull requests, debugging late-night issues, making architectural tradeoffs, and learning through failure.
More than anything, this is a story about learning by building something real and how two students trying to survive bad internet ended up building infrastructure they hope hundreds of applications can someday rely on.

Key Takeaways

The central idea of this talk is that performance is not something you add afterward. Fast systems usually emerge from handling failure well and keeping resources productive instead of idle.
Every meaningful optimization in Surge started with a benchmark surprising us. We would build a feature, measure it obsessively, notice behavior we could not explain, spend hours reading and experimenting, and only then change the system.
That process of building, measuring, questioning assumptions, and trusting data over intuition became the most valuable lesson from the project, one that applies far beyond download managers.
The talk also gives a practical look at how parallel downloading works at the HTTP layer. Most people know that more connections can improve speed, but the more interesting question is why naive parallelism often leaves bandwidth unused or even hurts performance.
We will explore the real engineering challenges behind download acceleration: coordinating workers, reacting to unstable network paths, recovering from slow connections, and recognizing when additional concurrency becomes overhead instead of optimization.
Much of Surge’s design emerged from studying unpredictable real-world behavior rather than following theory alone. Benchmark data repeatedly challenged our assumptions and forced us to rethink how networked systems behave in practice.
Beyond the technical story, this talk is also about the reality of maintaining open source in public. Building software alone is exciting; maintaining software people depend on is humbling.
Once users started relying on Surge, the challenges changed completely. We found ourselves reviewing pull requests, handling unexpected edge cases, thinking about stability and backwards compatibility, and confronting design flaws exposed by contributors and users.
Some of our hardest problems had little to do with downloading itself. We learned by breaking things, fixing them under pressure, and gradually realizing that open-source maintenance teaches systems thinking in ways very few university projects ever do.

References

https://surgedm.github.io

https://github.com/SurgeDM/Surge

Session Categories

Introducing a FOSS project or a new version of a popular project

Technology architecture

Engineering practice - productivity, debugging

Story of a FOSS project - from inception to growth

Talk License: MIT

Which track are you applying for?

Main track

Speakers

Meet Mehta Student | IIITG

Meet Mehta is a CSE student at IIIT Guwahati and co-maintainer of Surge. He is interested in low-level design, systems engineering, and open source.

Junaid Islam Student | IIITG

Junaid Islam is a CSE student at IIIT Guwahati, SEP Intern at JPMC, and co-maintainer of Surge. He is passionate about systems programming, problem solving, and open-source software.

Reviews

No reviews yet.