From Tensors to Talk: Inside the Architecture of llama.cpp

We, as software developers and power users, might have used tools like Ollama, LMStudio, Jan, and PocketPal to run large language models (LLMs). These tools run on a variety of platforms, ranging from compute-heavy Nvidia/AMD GPUs to lightweight 32-bit Arm CPUs and manage large chunks of memory distributed into physically different systems. The underlying component common to all of these tools that handles LLM execution is llama.cpp, an open-source model runtime that supports execution of different LLMs (like Llama, GPT, Qwen, etc.) across different platforms (macOS, Windows, Linux) and accelerators (Nvidia/AMD GPUs, Metal) for speeding up computations.

llama.cpp provides high-level APIs that tools like Ollama and LMStudio use to load different models and generate responses from the model. In this talk, we will deep-dive into what happens beneath the high-level and explore how llama.cpp, a LLM runtime, works as a system managing memory and computations.

We will discuss the following points:

• Revisiting model inference basics, a short intro on llama.cpp and its significance as an open-source model runtime.

• How llama.cpp loads a LLM model from disk into memory (host and device memory)?

• What is a LLM and how do we generate text given a query?

• Basics of computation graphs and how llama.cpp executes graphs efficiently by detecting operations that can be accelerated on certain devices.

The audience will understand how llama.cpp works internally and in-general how LLM inference is performed efficiently across platforms.

Students will be able to map fundamental concepts in CS and operating-systems to real-world applications in LLM inference, thus developing a concrete understanding of those concepts.