Breaking into the Black Box: Making LLMs Transparent for Science

What if we could peer inside the black box of LLMs and understand exactly how they reason through scientific problems? This session aims to turn around the notion of LLMs from mysterious neural networks into interpretable, debuggable systems using entirely open-source tools.

Unlike closed-source alternatives, we'll explore how the open architecture of models like DeepSeek and Evo-2 allow us to trace data flow, examine attention patterns, and understand decision-making processes at a granular level. We'll see how DeepSeek's reasoning pathways derive equations and how Evo-2's genomic knowledge can be interpreted to reveal cross-species correlations!

We'll be introducing FOSS evaluation frameworks like Promptfoo and Comet's Opik to systematically audit model performance and mechanistic interpretability tools like TransformerLens and Prisma to visualize internal representations and understand how these models process scientific concepts—from protein folding predictions to mathematical theorem proving.

This session bridges the gap between AI transparency and scientific discovery, showing how open-source interpretability tools can make LLMs accountable for research.

• Discover how open-source LLMs provide unprecedented transparency compared to closed alternatives, enabling scientific validation of AI reasoning

• Microscoping straight into LLMs with TransformerLens and Prisma

• Understand FOSS evaluation tools like Promptfoo and Opik to systematically assess LLM performance in research.

• Explore real scientific applications through models like Evo-2 for genomics and DeepSeek's breakthrough reasoning mode