Calibrate: A framework for continuously evaluating and improving AI agents within non-profits

AI agents have become ubiquitous and are increasingly becoming demonstrably good at a variety of tasks. Still, they make mistakes. I work with multiple non-profits across India and Africa through my time at Artpark and The Agency Fund on deploying AI agents in high-stakes domains like healthcare, agriculture and education where mistakes are costly. An example is show below.

Evaluation is key to ensuring AI deployments are reliable, safe and robust in these sensitive contexts. What does it mean? Simply put:

Does the AI system generate the desired output for all possible inputs?

As simple as it sounds, ensuring this in practice is a challenge. Very few people know how to evaluate AI agents well. As a result, they end up recreating the same evaluation harness for every new problem they work on, often with most common mistakes we see (e.g. using LLM judges for automated AI evals without aligning them to human preferences), which takes away precious time and resources which could have been spent on identifying and fixing real issues. The (few) existing tools are either too costly or too complex to use for non AI engineers. Moreover, most non-profits have scarce technical talent and must lean heavily on empowering domain experts (product managers, doctors, teachers, agronomists) to drive the evaluation.

In this talk, I will give a demo for Calibrate, an open-source AI agent evaluation platform for non-profits that I have built to empower domain experts to evaluate AI agents without writing code with evaluation best practices baked into every step.

Calibrate lets you:

• Unit test your agents: You can create “unit tests” that capture specific edge cases and check whether your agent responds appropriately. You can easily analyse its mistakes and iteratively test changes. Only deploy an agent once it passes all the critical unit tests.

• Use AI to evaluate AI: Statistical metrics fail to accurately measure the quality of open-ended responses produced by these models. For example, a metric like “Word error rate” (WER), commonly used to benchmark speech-to-text models relies on counting the mismatches between the predicted and reference transcripts. As a result, “9” and “nine” are treated differently. Calibrate lets you create LLM-based evaluators (LLM judges), which capture semantic meaning which makes evaluation more reliable, realistic and practical.

• Evaluate voice agents: Beyond text, you can evaluate voice agents as well by measuring the quality of the speech-to-text and text-to-speech models powering your agent.

• Find the best agent configuration: Identify the best speech-to-text models, text-to-speech models and LLMs for your use case on your datasets instead of relying on generic public benchmarks. Calibrate is vendor-agnostic and works with any model or provider stack.

• Align LLM judges to human judgment: LLM judges themselves can make mistakes. Before we employ them at large scale, they need to be “aligned” to human preferences. For domains like healthcare, there is often no single right answer and even experts might disagree. Calibrate makes it easy to collect unbiased human labels, measure consistency between human labels, track how human-aligned your LLM judges are, and iteratively fix them before trusting it for automatic monitoring at scale.

• Simulate realistic conversations before shipping: The features above let you “unit test” individual components of your agent (speech to text, LLM, text-to-speech) but the end user experiences your system as a whole. Errors from one component propagate to the next. Simulations are the “end-to-end” that help you capture mistakes of your agent as a whole. You can define user personas (WHO your end users are) and scenarios (WHAT are they trying to achieve) to stress-test your agent and catch regressions before real users are affected.

Calibrate comes as both a CLI and a webapp. Engineers might find the CLI to be the right interface for them and domain experts use the webapp. Both are fully open-source and the webapp is self-hostable so sensitive data stays on your infrastructure. Calibrate easily plugs into your CI pipelines through our Github action, preventing deployments that fail your evaluations.