Cloud hosted AI impose very real tradeoffs: namely inference latency, cost, security concerns as the data leaves the device, along with the resultant dependencies on proprietary APIs.
Improvements in model architectures, quantization techniques, and storage have enabled a credible alternative On-Device Inference
This talk will examines the different aspects of running models on Commodity Android hardware today.
A technical deep-dive into three production-ready runtimes:
whisper.cpp: how OpenAI's Whisper is ported to C++, its use of GGML for CPU-optimized tensor operations, and how it achieves real-time transcription on mobile CPUs
llama.cpp: the GGML/GGUF model format, quantization schemes, KV caching, and how the runtime maps computation across CPU cores and GPU backends
LiteRT-LM (formerly TensorFlow Lite): Google's approach to on-device generative AI, its delegate system for NPU/GPU acceleration, and how it differs architecturally from the llama.cpp stack
Beyond the individual runtimes, we'll look at what they share: memory-mapped weight loading, operator fusion, attention optimizations, and the design philosophy of keeping inference self-contained with no runtime dependencies.
The session closes with a live demonstration of an Android application I built performing inference fully offline. The pipeline demoed runs on a single loaded model (Gemma 4 E2B) with no reloads between stages:
Voice recording → Transcription → Mood detection → Calendar event extraction → ICS file → System calendar
No network, no cloud, no API keys. We'll look at the architecture of this app and the practical considerations for keeping it responsive on a mid-range device.
Some data measured on device:
App | | PSS delta | | RAM Used | | CPU | | GPU |
|---|
Demo App | | +1GB | | 754MB | | +11.2°C | | +10.1°C |
Clash of Clans | | +54MB | | 845MB | | +9.0°C | | +1.1°C |
Subway Surfers | | +21MB | | 915MB | | +6.0°C | | +2.5°C |
*Measured using ADB commands (max to min difference)
Operation | | Avg wall time | | Avg throughput | | Avg PSS delta | | Avg battery drain |
|---|
Engine init | | 4,858 ms | | NA | | +595 MB | | 0 µAh |
Transcription | | 10,217 ms | | 96.8 tok/s | | +338 MB | | 3,000 µAh |
Event extraction | | 5,009 ms | | 48.0 tok/s | | +60 MB | | 2,000 µAh |
*App metrics measured using Gemma 4 E2B on a OnePlus 13 device, processing 1–2 minute voice note clips (approximately 100–200 words of transcription).
Attendees will leave with a clear mental model of how these runtimes work under the hood, what the real cost dimensions of edge inference is on the device level, and what is genuinely feasible today on hardware most users already carry.
