– By Sudeep Gowrishankar, Head of Artificial Intelligence & Data, and Aparna Krishnakumar, Chief of Staff, Rocket Learning
1. Introduction to Shiksha Saathi
Over the last five years, Rocket Learning has trained 500,000+ Anganwadi Educators across 1 in every 3 communities in India, impacting over 7 million children. Through daily bite-sized content on WhatsApp, we have seen 75% of children reach school-readiness compared to a national average of 50%. This work has also given us an unfiltered view of what educators are up against. Most manage classrooms of 10 to 20 children below the age of 6, while simultaneously handling nutrition monitoring, government reporting, parent communication, and administrative tasks. Training support is limited, guidance arrives through paper manuals, and much of their work remains fragmented and time-consuming.
Rocket Learning’s core program has been a response to this reality. It includes daily lesson planning videos, weekly expert mini-trainings, and monthly in-person pedagogical training, all facilitated through digital communities of practice on WhatsApp. We have demonstrated through this intervention that Anganwadi educators enjoy using these content suggestions and training to implement standardized activities in the classrooms, and that this meaningfully moves learning outcomes.
However, we have had limitations – the lesson suggestions have all been the same for every classroom in a particular district, and for every child aged 3-6 in a classroom, rather than tailored for the particular levels of that classroom or the particular children in the classroom. While we have always guessed that more customized lesson plans will lead to greater learning outcomes, we had been constrained (pre the AI revolution) due to:
- the lack of detailed and ongoing student level learning data, and
- the enormous cost of generating millions of child-level learning plans (even if the data had existed).
For the first time, AI can give us solutions to these challenges. Shiksha Saathi is an additional layer supercharging Rocket Learning’s core program. It is a voice-based AI assistant, delivered on the same channel, designed to give Anganwadi educators and pre-school teachers in Rocket Learning’s program more agency to be quality educators. It does so in three ways:
Shiksha Saathi helps Anganwadi educators in three ways:
- Making sense of her day: She tells Shiksha Saathi what happened in classroom in her own words and in any order. It analyzes her notes to identify each child’s progress and her classroom’s needs, then generates a class report and a personalised lesson plan for the next day.
- Finding the right activity in the moment: She asks for a poem, story or activity that fits her need of the moment – adapted to the age of the children in her class, the materials she has available or the day’s learning objective – and gets one within seconds.
- Getting unstuck: When she needs help with government reporting apps, reporting rules, or any other type of technical support, it gives her a quick first answer and points her to the right resources for more in-depth information.
The purpose of Shiksha Saathi is simple: to help our Anganwadi educators spend less time on paperwork and preparation, and more time with children.
2. Why we evaluate, and what this post covers
Shiksha Saathi helps children only if the following conditions hold: the AI gives safe, relevant and accurate responses; educators find it useful enough to use it continuously; and it gives them more support and time for the teaching they already do, so children in their classroom learn better.
To check each of these, we follow the AI Evaluation Playbook developed by The Agency Fund, the Center for Global Development and IDInsight, which looks at four levels:
Each of these four levels asks a different question and needs different evidence. This post covers only the first level, “Does AI get it right?”. We will report on the other three levels in later blog posts.
3. What happens behind each reply
A reply from Shiksha Saathi is not the work of one AI model. Each message she sends, by voice, text or a button tap, passes through a short chain of steps, each handled by the model best suited to it. Because the steps are separate, we can test and improve each one on its own.
Every reply passes through a sequence of steps:
Two design choices matter for safety and accuracy. Shiksha Saathi turns what an educator says about her class into short, structured notes, and nothing is saved to a child’s record without a confirmation step. And it is deliberately narrow: it covers teaching, the curriculum, classroom notes and technical help, and does not give medical advice or counselling.
4. How we tested it
A mistake early in the chain can carry through to everything after it. If a child’s name is misheard at the start, it may be stored incorrectly in her classroom’s record, and the child could get an incorrectly personalised activity or report card. That’s why it’s essential to test each step on its own, and then all the steps together, to make sure that not only does each reply make sense, but the conversation as a whole goes well too.
At each step, we weighed accuracy, safety, speed, and cost. We compared the model we use today against newer and cheaper options from OpenAI, Google, Sarvam and others, using four kinds of checks:
1. Test sets: We created a test set of real messages from educators, each reviewed by hand by our team, who noted the correct answer for everything we wanted to check. In some cases, we changed a message slightly to cover situations that rarely come up in real conversations but that we still need to handle well. This gave us a set of messages where the right answers were already known, which we used to check how well we handled transcription, safety and routing.
2. AI judges: We used AI models from a different model family than the one powering the stage being tested. It reads our assistant’s replies and scores them against a written set of criteria we defined in advance. We chose a model from another company so it wouldn’t be inclined to favour replies similar to its own. For translation, note-taking and writing, we used two such judges and tracked how often they agreed. When two judges consistently give the same score, we can be more confident that the scores are reliable.
3. Human reviewers: Four members of our team who read 200 real conversations from October 2025, January 2026, and April 2026, checked what machines judge poorly.
4. Rule-based checks: simple, automated checks for logic-driven things like formatting.
5. What we found
5.1 Listening: hearing educators clearly, and quickly
Educators talk to the assistant through WhatsApp voice notes, often from a busy classroom. To process these requests, the assistant first runs a speech-to-text model that turns each voice note into text, and the rest of the pipeline then works on that text. We tested eight speech-to-text models on 1,000 voice notes in Hindi and Marathi. Sarvam’s models (we currently use saaras:v3) built for Indian languages and were among the most accurate and the fastest, replying in under half a second. But Marathi was consistently harder than Hindi, for every model we tried. This emphasizes a need to evaluate new models for every new language we adapt to.
Next: We are weighing a move to Sarvam’s newest model. As Rocket Learning expands to new regions, we will run the same tests to choose the right model for each language.
5.2 Understanding: speaking her language and knowing what she needs
Most of our content is in Hindi, so Shiksha Saathi translates it for educators working in Marathi or English. It then decides which of its three main use cases described above the educator needs in that moment. Our current model, GPT-4.1 mini, translated as well as the best alternative at about half the cost. It also routed messages correctly 88.5% of the time, the best of six models tested, and 96% to 100% of the time for the three main uses.
Next: We’ll keep this model, and re-test everything before moving to newer models as the older ones are being deprecated.
5.3 Screening out harmful messages
Every message an educator sends passes through two safety checks before Shiksha Saathi acts on it. Together, the combined checks stopped about four in five harmful messages. The harmful messages that got through didn’t match anything Shiksha Saathi is built to do, so it took no action on them and offered nothing harmful. Still, doing no harm is not enough: we want every such message recognised and handled as what it is.
Next: Teaching the second check to recognise unsafe messages directly and answer them with a safe, fixed reply.
5.4 Remembering: keeping every child’s story accurate
AI judges checked 1,890 notes the assistant saved about children against what educators actually said. It linked notes to the right child 96% of the time (correct attribution), and our reviewers found names and key details saved correctly in 94% of conversations (accuracy). Where it can do better is completeness: it sometimes misses something the educator mentioned.
Next: Having people check saved notes in more depth, so we know exactly what gets missed, and making sure it captures everything an educator shares more accurately.
5.5 Answering: finding the right content and writing a helpful reply
When an educator asks for a story, poem or activity, Shiksha Saathi writes one for her on the spot. Two AI judges scored 197 of these replies against a checklist, and they averaged 2.7 out of 3, the highest of the three models we compared. They were strongest on staying relevant to what she asked, reflecting good values, and being the right length. They were weaker on teaching depth and on suggesting materials she is likely to have at home. Looking more broadly, our reviewers rated 88% of all Shiksha Saathi’s answers as appropriate and something an educator could act on.
Finding the right content from our curriculum was the weakest step. When reviewers checked answers that relied on our curriculum, about one in four was built on material that wasn’t fully relevant to what the educator asked.
Next: We’ve improved our knowledge base and added a short guided flow that asks educators a few questions before searching. We’re continuing to explore technical and product improvements here.
5.6 Putting it all together
Two panels of AI judges, one using Claude models and one using Gemini models, each read the same 100 real conversations from start to finish. Safety held up well: personal data stayed protected, replies stayed within scope, and content stayed safe and inclusive in 92–100% of conversations. Activities came with complete instructions in 94% of cases and were always easy to implement in an Anganwadi. The weaker areas were keeping the wording simple, noticing when she is frustrated, and answering in-scope requests instead of declining them. The judges also disagreed most on these, which tells us the criteria need sharpening.
Next: Our early-childhood experts will help us improve the criteria the judges disagreed on, so that both judges score the same way a person would. We’ll then check a regular sample of live conversations automatically, and flag any that fail a safety check for our team to review.
6. What we learned, and where we’re going
At most steps, the models we chose held up well against newer and cheaper alternatives. Where they fell short, the evaluation identified which step and language needed revision. Four lessons stay with us:
- Look at each language on its own. Several steps worked better in Hindi than in Marathi, and averages would have hidden that.
- Test the whole setup, not just the model. Each step can pass its own test while the conversation as a whole still misses what the educator asked. That is why we also grade whole conversations, not only the steps inside them.
- Check the checkers. Our AI judges agreed least on teaching quality, the thing we care about most. In such cases, having a human to validate the AI judges becomes crucial.
- A test is a snapshot: Next, we want to move to live monitoring of the assistant by evaluating a sample of real conversations at regular intervals.
Next, we will better align our responses with pedagogy, and check our AI judges against Rocket Learning’s own early childhood experts. Beyond that, we want to learn whether educators keep coming back, whether Shiksha Saathi helps them teach the way they want to, and, in the end, whether the learning outcomes of children improve. These are stories for parts 2, 3, and 4 of this series of blogs on evaluating Shiksha Saathi.
If you are building or evaluating AI for frontline educators, especially in Indian languages, we would love to compare notes! Please reach out to us at sudeep.g@rocketlearning.org.
Contributors: Azeez Gupta, Badmavasan Kirouchenassamy, Balagopal KV, Burhan Ashraf, Manish Asodekar, Meenakshi Garg, Nishith Agarwal, Shivansh Nagi, Suresh Unnikrishnan