From "Batch Is Slower Than Sequential" to a 2.5× Speedup: Anatomy of GLiNER PR #292

In mid-2024, GLiNER's GitHub had a recurring complaint pinned in the Discussions tab: batch inference was slower than sequential. The root cause was the padding tax, common to any encoder model that batches variable-length inputs by padding up to the longest sequence. The fix is well known in research circles (sequence packing) but had not been wired into GLiNER's inference path.

This talk walks through how PR #292 happened end to end: reading the codebase cold, finding the right hook points in the forward pass, getting the patch merge-ready for v0.2.23, and the follow-up PR #301 that caught backbone-propagation bugs the first patch missed. The technical content (sequence packing on encoder models) is the vehicle. The real point is what is generalizable: how to ship your first upstream contribution to a research-led FOSS project without being a researcher yourself.

What the talk covers

1. The padding tax, in numbers. Why a batch of [10, 12, 15, 480]-token sequences spends roughly 73% of its FLOPs on padding. How the math gets worse as batch length variance grows. Why naive dynamic batching infrastructure makes this worse, not better.
2. Sequence packing, demystified. The mental model: concatenate variable-length sequences into one packed tensor, build a block-diagonal attention mask so each packed segment only attends to itself, unpack at the output. Where the technique came from (FlashAttention-2, HuggingFace ModernBERT), what changes for inference vs training, and why it lands cleanly on encoder-only models like GLiNER's DeBERTa backbone.
3. Reading the GLiNER source cold. Where the model lives, where the batching lives, where the data collator lives. Why the right hook points were not where I expected. How to navigate a 2.9k-star codebase you have never seen, with one stable branch and one researcher-maintainer who reviews on weekends.
4. PR #292, the actual diff. What the first version looked like, what the maintainers asked for, what changed before merge. The 2.5× speedup measurement: on what hardware, against what baseline, with what batch composition. What I would have done differently if I had known PR #301 was coming.
5. PR #301: the follow-up. What broke when other backbones tried to use the feature. How packing parameters silently failed to propagate through the call chain. The lesson about writing FOSS patches that survive future refactors.
6. The generalizable pattern. A repeatable checklist for shipping a first PR to a research-led FOSS project: how to scope, how to write the issue before the patch, how to read review cycles, what to do when the maintainer goes quiet, how to follow up without being a pest.

Why this matters for PuneFOSS

Research-led FOSS projects like GLiNER, vLLM, transformers, and llama.cpp are increasingly the infrastructure that production AI systems run on. They are also harder to contribute to than typical OSS projects, because the maintainers are usually one or two academic researchers with limited review bandwidth. This talk demystifies what it actually takes to ship into one of these projects, using one concrete contribution as the thread. No buzzwords, no AI hype: one PR, two commits, a 2.5× speedup, and a working pattern the audience can apply to their own first contribution Monday morning.

Prerequisites

Comfort with the idea of a transformer forward pass and attention masks. Python and PyTorch familiarity. No prior FOSS contribution experience required (that is partly the point).

Materials and links

1. GLiNER PR #292 (inference-time sequence packing): github.com/urchade/GLiNER/pull/292
2. GLiNER PR #301 (follow-up, multi-backbone fixes): github.com/urchade/GLiNER/pull/301
3. GLiNER serving documentation where the feature now ships as --enable-sequence-packing: github.com/urchade/GLiNER/blob/main/docs/serving.md
4. Public reproduction notebook with baseline vs packed timing on urchade/gliner_medium-v2.1

1. A working mental model of sequence packing and the padding tax it removes.
2. A reproducible benchmark methodology for batched encoder inference, runnable on a single GPU.
3. A checklist for shipping a first PR to a research-led FOSS project.
4. A line-by-line walkthrough of the public diffs (PR #292 and PR #301) that the audience can pull up on their phone during the talk.