Gesture-Driven Web Accessibility: Adapting Interfaces with Multimodal LLMs

What if the web could adapt to a user’s physical needs in real time?

In this talk, I present a gesture-driven accessibility layer built using MoveNet for real-time pose detection, powered directly in the browser with TensorFlow.js, and enhanced with multimodal LLM reasoning. MoveNet detects body keypoints efficiently on-device, while TensorFlow.js enables privacy-friendly, low-latency inference without server dependency.

A multimodal LLM interprets motion patterns in context — such as tremors, limited mobility, or repeated interaction failures — and refines user intent. The system then dynamically adjusts UI elements, improves focus management, and respects ARIA roles and WCAG principles. I’ll share architecture, safety considerations, and lessons from building an adaptive, AI-assisted web accessibility prototype.