Identifying User Competencies for Equitable LLM Adoption in Public Health

LLM assistants are often evaluated on whether they answer correctly, appropriately and whether people use them regularly. In public-good AI, with LLMs, the harder question is whether users have the competencies needed to work with the system: knowing what to ask, articulating the question, choosing an input mode, comprehending the answer, rephrasing or asking follow-up questions, and applying the response in their real work.

This becomes especially important when emerging users of technology interact with an emerging technology like AI. If the system silently demands skills that users have not yet built, even a technically strong tool can become hard to use, easy to abandon, or dependent on external support.

Drawing from my work as a Senior Design Researcher at Wadhwani AI on HealthVaani, an AI-powered assistant for frontline health workers, this talk shares a practical UX method for evaluating AI adoption through user competence. Using interaction logs, field research, interviews, and design probes, I will show how small behaviours such as relying on suggested questions, switching between voice and text, rephrasing queries, or dropping off after no answer can reveal where users are confident, hesitant, dependent, or stuck.

The talk introduces competence-based personas such as Influencers, Believers, Observers, and Onlookers, built from real usage patterns and support needs. For FOSS and public-good AI teams building LLM assistants, RAG systems, and multilingual tools, this offers a reusable way to evaluate whether AI systems are not just technically functional, but usable, recoverable, and equitable at scale.

1. Identify hidden user competencies required to use an AI system meaningfully.

2. Use logs, field research, interviews, and probes to understand user confidence, hesitation, dependency, and recovery.

3. Build competence-based personas to understand different support needs.

4. Design better onboarding, prompts, input support, examples, and recovery flows for AI systems at scale.