The energy bill of a prompt

What it is: Measuring actual joules consumed per generated token on real hardware, instead of guessing from parameter counts. Inference, not training, is now the dominant energy cost because it runs billions of times a day.

Why it matters: Cost (electricity is a line item for self-hosted LLMs), capacity planning for on-prem GPU servers, ESG/carbon reporting, and choosing models rationally - a quantised 7B may beat an FP16 3B on energy per useful answer.

The tools:

Zeus - precise GPU energy windows via NVML; prints J/token

greenToken - wraps any inference call, per-token J and CO2e, side-by-side leaderboard, pytest plugin to fail CI on regressions

CodeCarbon - simplest; CSV of energy_kwh, cpu/gpu power, emissions

LLenergyMeasure - research-grade across Transformers/vLLM/TensorRT-LLM

Inference, not training, is the main energy cost at scale, and almost nobody measures it.

Model size is a poor proxy. Quantisation, batching, and MoE sparsity matter more.

Batching is the biggest lever; J/token drops sharply as batch size grows.

Real numbers need hardware counters (NVML, RAPL). Token-based estimates can be far off.

Energy is a cost and capacity metric for on-prem GPUs, not just ESG.

Track J/token in CI like latency.