Open, decentralized language assessment software for the public school system

Summary

This talk describes an AI-driven, teacher-facing spoken language assessment software that is designed to be pedagogy-driven, privacy-respecting, for deployment in poorly-resourced* learning environments such as public school systems.

*'poorly-resourced' = poor/no internet, access to only commodity compute.

Why

The software aims to i) fill a gap in the traditional language assessment processes that are highly Writing-centric, and ii) demonstrate a model for fair and equitable Education Technology ('EdTech') design.

The technology is designed to support the following pedagogical requirements:

1. Sensitivity to early learners with home language / mixed language abilities
2. Support for monologic and dialogic oral productions (monolog is purely learner speech in response to a teacher's question; dialog is a free back-and-forth conversation between the teacher and the learner)
3. Teacher-configurable metrics

Technology stack (all components are FOSS)

1. A fork of the ODK Collect mobile app on Android to host assessment forms for offline use in the classroom. A fork was required to enable 100% offline use for form download from ,and submission uploads to, a local desktop computer
2. AI execution pipeline:
3. Pyannote for speaker separation (learner and teacher speech)
4. Whisper speech-to-text models for language identification and learner speech transcription
5. SpaCy (+ custom code) for NLP, a custom grammar model built on vennify.ai supported by the Errant library
6. Frappe-based custom desktop app as teacher front-end
7. Frappe Insights for visual reporting.
8. The 'KDE Connect' and 'LocalSend' mobile and desktop apps were used for offline data transfers between the ODK mobile app and the desktop app

Tech design highlights

1. Ability to detect highly indistinct, hesitant learner's speech (disfluencies) with reasonabl accuracy
2. Intentional design to minimize RAM usage to under 6 GB to support commodity 8GB RAM desktop/laptops
3. Ability to work well with typical classroom/surrounding noises - mainly thanks to Whisper's robustness. Tested in real-world government school classrooms in Bengaluru.

Do-no-harm design considerations (mainly to minimize imperfect transcriptions and speech separations)

1. Incorporates minimum thresholds for reporting
2. Reports aggregates and broad trends over the academic year, not individual performances in narrow timescales

Roadmap (coming year 2026-27)

1. Expand prototype deployment and use in more schools
2. Optional qualitative assessment of contents of speech using a local SLM
3. Develop more meaningful, actionable insights for teacher (patterns of speech productions, clusters of learners at similar levels, recommended learning resources/activities for different learner levels)

1. Attributes of a fair and equitable Education Technology ('EdTech') design.
2. Some ideas for fully local design (hopefully) and typical challenges in the public school system