From 4B to 1B: How I Built a Phone-Ready Legal AI for India with Open LLM(Gemma)

India has over 1.4 billion people and almost zero accessible legal AI. Most LLMs don't know the difference between IPC and BNS, hallucinate section numbers, and were never trained on Indian law. I built LexiMini to change that — an open-source legal AI fine-tuned specifically on Indian law: BNS, BNSS, tenant rights, labour law, women's protections, and constitutional provisions.

This talk is the full engineering story — not just what worked, but what broke. I'll walk through curating 400+ Indian legal entries into 3800 training samples, fine-tuning Gemma 3 4B using MaxText on TPU, then distilling that knowledge into a 1B student model using Google DeepMind's Tunix framework — small enough to run locally via Ollama on a consumer machine.

I'll be honest about the hard parts: hallucination on rare legal sections, the tradeoffs of logit-based distillation (temperature=2.0, alpha=0.7), and why JAX tooling is powerful but poorly documented. You'll leave with a reusable open-source blueprint for domain-specific model compression — applicable to any Indian language or legal domain.

• How to curate and structure domain-specific legal data for LLM fine-tuning

• Step-by-step pipeline: Gemma 3 4B fine-tuning with MaxText on TPU

• How logit-based knowledge distillation works (4B teacher → 1B student) using Tunix

• Real tradeoffs of compression — what you gain, what you lose

• A reusable open-source blueprint applicable to any Indian language or domain