Evaluating AI Universe for your Dialects and Languages: Methodology for Architecting Low-Resource Language Evals

Many of Asia and Africa’s most widely spoken languages have historically been “invisible” to AI due to limited multilingual datasets used to train LLMs. Ideally, everyone regardless of their language or location should be able to access digital services through voice, not text. Organizations seeking to operate AI in these  environments each wrestle with a remarkably similar set of challenges:

• Model Selection: Which models actually understand users’ languages?

• Rapid Prototyping: How do you get to a working voice, WhatsApp, or Android solution fast with low latency?

• Continuous benchmarking: As new models drop weekly, how do you know if they're better for you?

• Sovereign deployment: How do you run models inside your own country's infrastructure, affordably?

• Avoid re-inventing the wheel: Can orgs avoid creating cumbersome, expensive custom eval pipeline and just fine-tune on top of existing pipelines

RQ: If a user asks a spoken question in low resource languages like Swahili, Kannada, or Bengali, can an AI system understand it and provide a correct answer?

Gooey.AI has developed a consistent methodology around audio inputs, expert-provided answers (Golden Q&A) and LLM-as-a-judge evaluations. This talk will walk through  

• Gooey's Golden Q&A evaluation architecture for private + open-source models on digitally underrepresented languages

• Highlight real-world use cases with Gates Foundation of health, agriculture and education audio questions and determine how well and quickly different model combinations respond to these questions. 

• Highlight how orgs can create audio evaluations with their own data for better accuracy on regional dialects, code-switching formats etc

• Examples of our work from Gates Foundation, Rockefeller Foundation, Wellcome Trust, British Council, UN IOM, City of Seattle

• Current landscape and ecosystem progress of AI models in low-resource langs

• The efficacy of real-world, contextual evaluation model benchmarks that cater to global languages and dialects

• Golden Q&A and LLM-as-judge methodology as replicable and open mechanism for model evals

• DPG infrastructure to eliminate redundant technical work across the social sector globally

• Counteract historical inequities in AI development for low-resource langs: ensure productive benefits of AI are not concentrated in global north, english speaking communities