Study 01 · 2026-07-13
Does modelling the links beat scoring words one at a time?
The question
Vaaani's central premise is that a child knows a word when the links around it hold — its sounds, spellings, morphemes, cognates — not as an isolated score per word. If that is true, then for a word a child has never practised, the state of the surrounding links should predict success better than the word's own score.
How we tested it
Learners practise a connected region of the real engine graph, the way a curriculum actually moves. Their true competence on unpractised words comes only from what transfers across links, at a strength we control. We then ask four estimators to predict success on 80,000 held-out words nobody practised: standard item-level knowledge tracing, a node-neighbour average, Vaaani's link belief, and the percolation gate.
What came out
Item-level knowledge tracing — the industry standard — is at chance (AUC 0.500) for predicting transfer to an unpractised word, even with prerequisites added. Modelling the relations lifts it to AUC ≈ 0.58, and the lift grows the more connected the child's knowledge is. When we set transfer to zero, every estimator drops back to 0.500 — the harness is not manufacturing an effect.
What it does not prove
Vaaani's link-level tracing ties a simpler baseline that just averages a node's neighbours (difference ≤ 0.003) — both read the same local neighbourhood, so we cannot claim link-tracing predicts transfer better than neighbour-averaging. The binary percolation gate (0.53) is a worse predictor than the continuous signal (0.58); it earns its place as a mastery gate, not as a forecast. An earlier version of this harness leaked a spurious +0.04 through its own generator; we found it, fixed it, and left it in the write-up.
Study 02 · 2026-07-13
When a child misses a sound their home language lacks, does the engine blame the right thing?
The question
Bengali has no /z/, so “zero” arrives as “jero”. Hindi merges /v/ and /w/; both lack the “th” in three. Contrastive analysis predicts a learner maps a missing sound onto the nearest one they own. Vaaani encodes each as an inhibitory link. The test: when a child misses such a sound, does the cause-net attribute it to the home language — and stay quiet for sounds the home language shares?
How we tested it
Sixty learners per home language (Bengali, Hindi). Force a miss on every predicted-confusable sound and every shared sound, run the real diagnosis code, and measure how cleanly the home-language explanation separates the two groups.
What came out
A missed /z/ for a Bengali learner is attributed to the home language at 0.45; a missed /m/ at 0.045 — clean separation for both languages. This is what makes per-sound home-language diagnosis possible at all: the earlier heuristic could never fire, because it looked for a script signal that phoneme nodes do not carry.
What it does not prove
The separation is perfect because the attribution is structured — it is not a noisy prediction being scored. This is not a claim about accuracy on recorded speech. The confusion weights come from the phonetics literature and still need calibrating against real learner data, and the substitution is currently inferred from a miss on the target sound rather than observed directly.
Study 03 · 2026-07-13
Does “knows it on paper, but not in the moment” predict what happens next?
The question
The fifth instrument reads how an answer arrives — fast and sure, or slow and self-corrected. Is the gap between knowing a rule and using it a real predictor of what a child retains, or just a plausible story?
How we tested it
Three hundred generated learners, each with two independent hidden traits: competence, which drives whether today's answer is right, and automaticity, which drives how long it takes — and, when we switch it on, how much competence survives to a future item. Response times carry realistic noise, because a classroom is full of reasons a child pauses. We sweep the effect from zero upward, so the null case is tested first.
What came out
With the effect switched off, fluency-weighting adds nothing (+0.001, confidence interval spanning zero) — the method does not invent signal. Where automaticity genuinely governs retention, it helps, and the fluency label does stratify who slips later.
What it does not prove
The improvement in ranking accuracy is small — about 0.005 AUC. Correctness already does most of the work; fluency re-ranks the ambiguous middle, the competent-but-slow cases. We can defensibly say Vaaani flags who is likely to slip, with reasoning you can read. We cannot yet say it measures a learning gap as a calibrated instrument with known error bounds. That needs real classroom data.
Study 04 · 2026-07-13
What is the fastest route through a child's graph?
The question
If mastery is about connecting knowledge, the best teaching order is the one that connects a child's graph fastest. Should the engine teach the easy links, the structurally important bridges, or something in between?
How we tested it
Run competing sequencing policies over the same graph and measure how quickly each connects it — area under the coverage curve, how much of the graph is reached in the end, and how many steps to get halfway.
| Policy | Area under curve | Final coverage | Steps to 50% |
|---|
| Learnable bridges (hybrid) | 0.247 | 0.533 | 240 |
| Pure bridge (betweenness) | 0.236 | 0.517 | 250 |
| Curvature — the shipped policy | 0.206 | 0.394 | — |
What it does not prove
This is a routing result, not a classroom outcome — connecting a graph faster is not the same as a child learning faster, and that link is untested. We are also publishing a result that goes against us: the policy currently shipping is the weakest of the three on this measure.