Sovereign Tokens: Running Production Grade Open-Weight LLMs Yourself

The Reality Check:

This field report details the technical architecture and implementation strategies used to deploy self-hosted LLM infrastructure. Drawing directly from the production stack operated at the Centre for Smart Governance, Government of Karnataka, this session focuses on the critical engineering decisions required when transitioning away from third-party APIs to fully owned, on-premise hardware

What We’ll Cover:

• The Blueprint: Choosing the right models and navigating the field of quantization.

• The Engines: Why ollama, vLLM and SGLang exist, where they break, and exactly what each is best at.

• The Bare Metal: How we configured vLLM on a single H200, including the specific six flags that gave us our biggest performance wins.

• The Minimal Production Stack: Keeping it lean with just a gateway, an engine, and observability plus the four metrics we monitor on every single release.

• The Payoff: How an OpenAI-compatible API turns your self-hosted gateway into a drop-in replacement that instantly plays nice with tools like opencode, OpenWebUI, Zed etc.

Who This Is For:

1. The Indie Dev who is tired of eye-watering API bills.

2. The FOSS Contributor trying to figure out which inference engine is worth betting their time on.

3. The Enterprise/Gov Architect handling data that legally cannot cross national borders.

Same stack, completely different stakes.

What You’ll Leave With:

You’ll walk out with our exact production vLLM configuration, a battle tested recipe for serving LLMs, Speech-to-Text (STT), Neural Machine Translation (NMT), and embeddings all on a single GPU, and the concrete steps needed to point your favourite code editor at your own sovereign stack that very evening.

Announcing a community driven initiative

Beyond this session, we are formally teasing the launch of an open-sourced, community-driven platform designed to decentralise self-hosting knowledge. This platform will host a running handbook detailing precise deployment strategies tailored to specific hardware configurations. This handbook is will be a guide for developers (irrespective of their knowledge in inference) to setup and host open weight models.

Additionally, a comprehensive benchmarking suite is currently in development. This tool will empower the community to publish their unique infrastructure setups alongside verified, real-world inference performance data, establishing a transparent repository for independent hardware optimization.

Key Takeaways

• Data Sovereignty Thresholds: Understanding the operational transition point where data exposure shifts from an individual privacy risk to an organisational sovereignty violation based on daily telemetry and prompt data leakage.

• Inference Tier Classification: A clear architectural distinction between local/personal runtimes (ollama, llama.cpp) and multi-user production engines (vLLM, SGLang) to prevent misaligned deployments.

• Production vLLM Configuration: The exact optimization profile used in production, detailing the six specific configuration flags implemented on an NVIDIA H200 to maximize throughput and latency efficiency.

• The Baseline Production Stack: The blueprint for a minimal, resilient architecture containing only three non-negotiable components: an API gateway, the inference engine, and the observability layer.

• Core Performance Metrics: A deep dive into the four critical vectors required to monitor and tune production health: Time-to-First-Token (TTFT) p95, generation tokens per second, GPU/HBM memory utilization, and request queue depth.

• Gateway Standardizations: Practical instructions on using an OpenAI-compatible API layer to serve as a universal adapter, connecting frontends like opencode and OpenWebUI with just two configuration values.

• Infrastructure Decision Framework: A repeatable, workload-driven matrix for matching model selection, quantization levels, and hardware specifications to actual institutional needs rather than vendor benchmarks.