Privacy First: Building LLM-Powered Web Apps with client side WASM

It’s no secret that for a long time machine learning has been mostly a Python game, but the recent surge in popularity of Gen AI has brought many new developers into the field. With JavaScript being the most widely-used programming language, it’s no surprise that this has included many web developers.

There’s been a ton of ink spilled on building with LLMs via API calls to the likes of OpenAI, Anthropic, Google, and others but in these cases, the user is sending the data and the prompt to the servers of these tools and hence is not a 100% secure. and raises issues like cost, latency, and privacy. Moreover certain use-cases might require companies to run models on Device

This is where open source tools like WebLLM, Transformers.js come into the picture. We detail constructing a complete language model pipeline that runs fully in the browser. such has interfacing with a state-of-the-art model like Gemma.

By using lightweight packages like Transformers.js we can quantize and compile models to run efficiently on the user’s device all thanks to WASM. This allows us to build complex conversational workflows like retrieval augmented generation entirely on-device.

Ecosystem

There’s been a ton of ink spilled on building with LLMs via API calls to the likes of OpenAI, Anthropic, Google, and others but in these cases, the user is sending the data and the prompt to the servers of these tools and hence is not a 100% secure. and raises issues like cost, latency, and privacy. Moreover certain use-cases might require companies to run models on Device

Thus this talk covers such use-cases and shares real world cases for on browser AI adoption for modern Web Apps while covering the technical architecture of building such apps