Devnagri Black Bird tops Voice of India benchmark across 15 Indian languages

Devnagri AI said its new speech recognition model, Black Bird, posted the lowest error rate on the Voice of India benchmark across 15 Indian languages, outperforming Google Gemini 3 Pro and Sarvam Saras V3. The result matters for banking, government and commerce use cases where accurate phone-based transcription in local languages can reduce workflow errors and compliance risk.

Why it matters: - Devnagri Black Bird’s performance points to a higher bar for Indian-language speech recognition in real-world phone calls. - Better accuracy can improve customer onboarding, collections, support, renewals and grievance handling in banking, financial services and insurance. - Lower transcription errors can reduce failed workflows, repeat effort and compliance risk. - The result also matters for government helplines and digital commerce, where many conversations happen in regional languages on noisy lines.

What happened: - Devnagri AI said its new automatic speech recognition model, Devnagri Black Bird, led the Voice of India benchmark. - The benchmark was created by AI4Bharat at IIT Madras in collaboration with Josh Talks. - Devnagri Black Bird recorded an average Open Indic Word Error Rate of 8.7, the lowest among the systems evaluated. - The company said the model ranked first across all 15 Indian languages in the benchmark.

The details: - Devnagri said the 8.7 score was less than half of Google Gemini 3 Pro’s 21.1. - Devnagri’s September 2026 evaluation said Black Bird was about 19% better than Sarvam Saras V3 at 10.7. - The same evaluation put Black Bird about 59% ahead of Gemini 3 Pro and about 62% ahead of Gemini 3 Flash at 23.1. - Devnagri said Black Bird also cut error rates by about 25% versus its earlier Shivani ASR model. - Voice of India launched on April 21, 2026 to test speech technology on how people in India actually speak. - The benchmark uses 536 hours of unscripted, real-world telephone conversations in 15 Indian languages. - The dataset includes 306,230 utterances and 36,691 native speakers, balanced by gender and age group. - Unlike scripted test sets, Voice of India captures noise, regional accents and code-mixing from everyday calls. - The benchmark accepts valid spelling and transliteration variants instead of treating natural language variation as errors. - Devnagri said Black Bird was built for the same environment, including code-mixed speech, regional accents and cluttered telephony. - The model is positioned as an Indian-language speech intelligence layer for contact centers and voice-agent workloads. - Devnagri evaluated nine systems using the same method. - Each system was tested through its production API under the same default settings and scored against the same references. - Devnagri kept the test set private to reduce overfitting and retained model versions and logs to verify results. - Results for other competitors were measured using publicly available production APIs and should not be treated as official vendor-published results. - Black Bird led in all 15 languages, including lower-resource languages such as Bhojpuri and Maithili. - Devnagri said the lead was not limited to high-resource languages like Hindi. - The company said the model outperformed Google Gemini 3 Pro and Sarvam Saras V3 on the benchmark.

Between the lines: - The benchmark is designed to reflect spoken Indian language, not clean lab audio, so the ranking is more relevant to enterprise voice deployments. - In enterprise voice AI, speech recognition is the first layer, so early errors in names, dates, amounts or intent can cascade through the rest of the workflow. - The result underscores a broader shift from generic multilingual models toward systems tuned for Indian code-switching, accents and noisy phone environments. - Devnagri is using the benchmark win to position itself as a specialist in regulated-industry voice infrastructure rather than a general-purpose AI vendor.

What's next: - Devnagri is likely to continue iterating on Black Bird as it targets enterprise deployments in banking, government services and digital commerce. - The company is also signaling a push to define India-specific performance standards for speech AI based on live telephone use. - As more workflows move to voice and phone channels, benchmark leadership could become a key sales signal for enterprise customers.

The bottom line: - Devnagri Black Bird now has a public benchmark lead that strengthens its pitch for real-world Indian-language voice automation.