With the field of on-device AI picking up steam, there is a gap between how on-device AI gets built and the kind of devices it may end up running on. Models are usually developed and evaluated on a handful of well-known machines, but they ship into a vast, diverse ecosystem of laptops and desktops, and developers may not have access to each such device.
A single trained model can reach a user's machine through several paths: CoreML or CoreAI on Apple Silicon, ONNX Runtime, Windows DirectML, OpenVINO, running on a CPU, an integrated or discrete GPU, or an NPU. Each of these paths does operations like conversion, casting, fusing differently which naturally leads to outputs that differ between backends and performance that differs from what was measured on a development machine while rolling the model out / vetting it for release. Yet addressing this amount of comprehensive benchmarking often goes relatively unaddressed. Devs usually don’t have devices lying around and co-ordinating testing across them may require support from those willing to volunteer their devices while also being tedious routine work due to the absence of any such harness in the space.
This talk looks at how to check that an on-device model behaves consistently, and performs acceptably, across that diversity of end-user environments. It maps the current desktop runtime and hardware landscapes, the common sources of numerical divergence across backends, ways of setting parity tolerances, the factors that make on-device benchmarks misleading and a hardware-in-the-loop CI one can use to run these models on a matrix of real devices and backends, checking parity and performance together before release.
The examples come from shipping small computer-vision models for image editing to a varied population of user machines.