Evals First, Code Later: A Practical Guide to Evaluations, Rerankers & Caches

Most RAG and search systems "just work"... until they silently fail in production.

I learned that the hard way. We were indexing millions of lines of code. Average latency was 20 minutes, relevance was weak, and most issues only showed up after shipping. It was painful.

So in order to fix this, we completely flipped our dev process, started treating evals like tests, and not afterthoughts. That led to a 40x latency drop, 30% relevance boost, and even the indexing time cut down to mere 30s.

This talk is a practical walkthrough of what worked for us:

• Setting up Evals Pipelines

• Combining BM25 with ANN retrieval

• Using re-rankers.

• Semantic caching strategies that keep results fast and fresh
  

We’ll look at:

• Performance benchmarks before vs after

• A live demo using Postgres (pgvector) + rerankers

• A cheatsheet + open-source repo so you can plug this into your own stack

Whether you're just getting started or already building with LLMs, search, or RAG, this will give you concrete techniques to improve relevance, speed, and reliability.

You'll be able to iterate based on systemic methods, not on intuition or vibes.