OpenSource Models as a Case Study for Alignment

Most people treat alignment in LLMs as a solved problem. The model refuses, so it must be safe, right? This talk challenges that assumption by going a level deeper.

I'll walk through how I used representation engineering and mechanistic interpretability to surgically remove safety guardrails from open-source models, layer by layer. The technique (based on Arditi et al.) works by identifying the "refusal direction" in the residual stream and projecting it out. What you get back is a model that behaves exactly as it did before training-time alignment. And that gap tells you a lot.

I ran this on three models: GPT OSS 120B, Sarvam 30B, and Sarvam 105B MoE (the largest Indian LLM to date). The results are published on HuggingFace at aoxo/sarvam-30b-uncensored and aoxo/sarvam-105b-uncensored, and the process was fully open-source throughout.

The core question this talk answers is: when you peel away the RLHF and instruction tuning, how aligned is the base model really? For some of these models, the answer is "more than you'd think." For others, not so much. We'll go through what the internal representations look like before and after ablation, how the refusal geometry changes across layers, and what that tells us about the robustness of the alignment approach used.

We'll also look at how Anthropic has approached this differently at the architectural level, and why their alignment holds up better under this kind of probing.

Tools used: TransformerLens, HuggingFace Transformers, and a few custom hooks. All open-source, all reproducible.

• How representation engineering works under the hood and why it is a more principled way to probe alignment than prompt injection

• What abliterating each layer of a model actually shows you about where safety behavior is stored vs. generalized

• A comparative look at how Sarvam 30B and 120B MoE hold up under this kind of interpretability audit, versus GPT OSS 120B

• Why "the model refuses" is not the same as "the model is aligned," and how to tell the difference

• How to run these experiments yourself using open-source tools on any open-weight model