Developing safety guardrails for AI use cases in the Indian social sector

The problem statement 

Foundational models like Claude, GPT, and Gemini can read and write Indian languages fluently but fluency does not ensure safety. In real deployments serving health and education needs, these models miss the sociocultural norms, colloquial usage, and shared context that grassroots communities live in. This results in subtle, repeated harms that standard model evals never surface. This talk shares how we found those harms across 4 live NGO deployments and built Kaapi Guardrails — an open-source, configurable safety layer — to catch them.

How it looks in real scenarios

When AI tools go live in the social sector, the failures aren't hypothetical — they're specific and recurring:

• A user shares a phone number, Aadhaar number, or a PDF medical report in-line, and it flows straight through to the model.

• A health query gets a response carrying a caste/gender slur or an unfounded gender assumption ("Ask your doctor about these symptoms, he can guide you…").

• A user describes symptoms to a maternal-health information bot, and instead of pointing them to a clinic, it confidently prescribes a specific medicine and dosage. Thereby, providing medical advice it was never built, qualified, or verified to give. 

These are the gaps between "the model speaks Hindi" and "the model is safe for this community.

The solution

We collaborated with 4 grassroots NGOs across health and edtech —  SNEHA, VigyanShaala, TAP, Inqui-Lab — to study their on-ground AI deployments, surface immediate safety risks, and build guardrails customisable across domains.

Primary research pointed us to 6 priority issues: PII removal, slur detection, topic relevance, toxicity detection, gender-assumption-bias removal, and answer relevance.

Our Evaluation Method

Manual annotation of input–output pairs sampled from pilot and live data, labelled with domain-specific labels and analysed for patterns of repeated error:

• SNEHA (Healthcare): ~60,000 pairs → 1,000 annotated (~1.6%)

• VigyanShaala (EdTech): ~20,000 pairs → 500 annotated (~2.5%)

Regular client consultations: The annotation exercise was supported by regular client consultations to understand how their end users interact with their AI tool, which included the following: 

1. Gathering information on actual usage of their AI tool versus its intended purpose.

2. Risks and harms they anticipated and were observing during pilot runs. 

3. Providing recommendations for safety design based on the manual evaluations. 

4. Discussing configuration needs for individual safety guardrails through multiple rounds of testing on live data.

Technical implementation of guardrails 

A standalone REST microservice that sits between your app and the LLM. You call it before sending a prompt to the model (input guardrail) and after receiving the reply (output guardrail); each call returns a safe/unsafe verdict and, optionally, a redacted or rephrased version of the text.

• Stack: FastAPI + SQLModel + PostgreSQL, Dockerised; validation engine is the open-source guardrails-ai SDK (v0.10.0).

• Configuration-driven, not code-driven: validator presets, banned-word sets, and topic-scope definitions are stored in the DB (scoped per organization_id) — NGOs tune safety to their domain without touching code or redeploying.

• Multitenant + observable: every call writes a request_log, and each validator writes a validator_log (PASS/FAIL, input, output, error). User input is redacted from error logs by design, so observability never becomes a leakage vector.

The validator inventory (input → output)

Pick the right response per risk — three on-fail modes: fix (redact/correct silently), exception (block the message), or rephrase (signal the caller to ask the user to rephrase). Not every unsafe input should be handled the same way.

It works (before → after)

Fig. 1. AI interaction with Guardrail 

1. Manual evaluation is critical for surfacing nuanced multilingual/multicultural risks that emerge over time. Remember model fluency in a language is not the same as safety in its culture.

2. Comprehensive manual evals include golden-dataset review, pilot + live annotation, and review of guardrail outputs.

3. Manual + automated evals must run in tandem, iteratively, to reach a robust safety design for your specific use case and threshold.

4. Make safety configuration-driven, not code-driven. Severity levels, banned-word sets, topic scopes, and bias categories belong in DB-backed config referenced by ID — so each NGO adapts guardrails to its domain without redeploying.

5. Measure the precision/recall trade-off, and latency, before trusting a guardrail live.