On-device inference: deploying and debugging models on Android

As AI powered applications become more common, there is a growing concern for the cost, latency, privacy and dependence on proprietary and expensive models hosted in the Cloud. At the same time advances in hardware make compelling solutions for alternative solutions namely: inference over the edge.

One device inference at the edge, involves running a model directly on client hardware without involving the network. What was once considered impractical is now achievable on commodity smartphones.

This shift has been enabled by a new generation of production ready runtimes, including whisper.cpp for speech processing, llama.cpp for local language model inference, and LiteRT-LM for deploying generative AI workloads on-device. This talk explores all three in depth.

We will also discuss the shared foundation and philosophies in the runtimes, the problem each solves along with a look at deeper architectural concepts

The session concludes with a live demonstration of a fully offline Android application running Gemma 4 E2B locally. The application records a voice note, transcribes it on device, extracts calendar events using a local language model, and creates the event in the system calendar all without network access or cloud services.

1. What ggml does differently from PyTorch / TensorFlow
2. A working sense of quantisation trade-offs
3. The real cost dimensions of edge inference (beyond the model size, including but not limited to: KV Cache Optimization and Cold Starts)