AI Without the Cloud: Privacy, Quantisation, and Local Inference Explained

Every time we send a prompt to ChatGPT or any cloud LLM API, our data leaves our device, travels to a server we don't control, gets processed by a company whose data policies we've probably never read, and might be used to train next model. For most casual use that feels fine, until it isn't especially for medical records, legal documents, proprietary codes and personal conversations. By 2025, 44% of organisations identified data privacy and security as their top barrier to adopting LLMs. What's surprising is that we've this solution since 2023, completely open source, runs on our laptop without a GPU and most developers still haven't heard of it.

This talk is about llama.cpp: an open-source C++ runtime for running large language models locally, with zero external dependencies, created after Meta released LLaMA weights in early 2023. It's also about quantisation, the technique that takes a 140GB model and makes it small enough to run on a ₹40k laptop or in my case, a phone. It's about understanding what we're actually trading away when we pick a cloud LLM over a local one and how to make that decision consciously instead of by default.

I'll cover quantisation from start, then look at what we lose moving from Q8 to Q4 or Q2, and finish by running a model live on a laptop. I ‘ll cover the GGUF format, compare llama.cpp against Ollama and vLLM (not just benchmarks but when to actually pick each one), and talk about vendor lock-in, what it costs you, how quietly it happens and how local inference breaks that dependency entirely.

The talk draws from real experience where I have used llama.cpp in a hackathon to deploy a model running fully on-device on a phone, with no internet connection, no API key, and no data leaving the device. That project won.

By the end you won't just know how llama.cpp works, you'll know whether it's right for you, how to get started without a GPU and why running AI locally is one of the most practical things you can do right now for your privacy, your costs, and your independence from big tech infrastructure.

1. Stop overpaying for AI you don't control and start understanding what cloud LLM dependency is actually costing you in terms of privacy and money.
2. Every GGUF label on HuggingFace will finally make sense to make informed choices instead of downloading whatever looks right.
3. Know why llama.cpp, Ollama, and vLLM are built for completely different jobs and how picking the wrong one has been quietly slowing you down.
4. Understand what local inference genuinely protects you from and just as importantly what it doesn't because that gap is where most people get it wrong.
5. Leave with a working setup to run a real capable LLM on your laptop without a GPU, a paid API, or a cloud account.