MicCheck Case Study

MicCheck by KaryaBeyond clarity: what makes voice AI sound human

MicCheck by Karya ranks today's leading AI voices across nine Indian languages, judged by the people who speak them. A cohort of trained native-speaker evaluators compared and rated commercial and open-source systems in two listening studies, using a six-question rubric built for them. Every voice was tested on sentences drawn from real questions people ask about agriculture, healthcare, education and finance.

9Indian languages
9Text-to-speech systems
>70,000Evaluator judgements

Listening studies conducted September to October 2026.

Leaderboard

Rankings by language and domain, decided by native-speaker evaluators.

Reading the leaderboard
  • Rank: systems in order of their Comparative score.
  • Type: commercial or open-source.
  • : Indian or international provider.
  • : each system's strength from the Comparative study. Higher is better, and 0 is the average.
  • Win rate: the share of all head-to-head comparisons a system won, shown for reference. The ranking uses the Comparative score, not win rate.
  • : each system's average rating in the Standalone study, across its six dimensions. Only the seven systems in the Standalone study have one.
  • Languages tested: how many of the nine languages each system was evaluated in.

Audio generated 26 August to 2 September 2026. Comparative study conducted 2 to 22 September 2026; Standalone study conducted 28 September to 26 October 2026.

Indian Language Coverage by Commercial TTS Providers (Updated Nov 4, 2026)

Provider Model type Indian English Hindi Bengali Tamil Telugu Kannada Malayalam Marathi Gujarati Punjabi Odia Urdu Assamese
ElevenLabsEleven v2 Neural (transformer) Yes Yes No Yes No No No No No No No No No
ElevenLabsEleven v3 Neural (transformer) Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Smallest AILightning v3.1 / Pro Neural (low-latency) Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No
Sarvam AIBulbul v3 Neural + LLM text norm. Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No
Microsoft AzureNeural TTS Neural Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
GoogleGemini TTS LLM-native audio Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Google CloudWaveNet/Chirp3-HD Neural (WaveNet/transformer) Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes No
Amazon Polly Neural (+ legacy concatenative) Yes Yes No No No No No No No No No No No
OpenAIgpt-4o-mini-tts LLM-based Yes Yes No No No No No No No No No No No
DeepgramAura-2 Neural No No No No No No No No No No No No No
CartesiaSonic 3.5 Neural (state-space model) Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No No
RimeCoda Neural (LLM backbone) Yes Yes No No No No No No No No No No No
Acapela Group Concatenative + neural Yes Yes No No No No No No No No No No No
Camb AIMARS8, commercial API Neural Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No No
Maya ResearchMaya 2 / Maya-2-Native Neural Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No
xAIGrok TTS API Neural Yes Yes Yes No No Yes No No Yes No No No No
MiniMaxSpeech-2.8 Neural (autoregressive) Yes Yes Yes Yes Yes No No Yes No No No Yes No
ReverieRevUp Voice API Neural Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes
Gnani.aiVachana TTS Neural (voice cloning) Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes* No Yes
YesNo* Gnani.ai, Odia: yes, but SDK not supported.

Not every provider offers every language, so Odia has eight systems and Urdu six in the Comparative study, and six and four in the Standalone study. Six systems are available in all nine languages, and they link results across languages in the overall view.

Versions and voices tested. For each system, we chose comparable voices across all languages, to the extent each provider's options allowed.

Key Findings & Takeaways

  1. 01
    Finding

    Clear leaders emerge overall, but the best voice varies by language.

    Cartesia, Google Gemini and ElevenLabs lead overall, and every language has a clear top tier. Smallest AI is among the leaders in Hindi and Indian English. Domain barely matters, although numbers in finance content are flagged as hard to follow.

    Takeaway

    Choose by language, then by use case.

    Where several systems share the top spot for a language, deployers have more strong options to pick from – test these with native speakers on your specific content and use case to find the best fit.

  2. 02
    Finding

    With accuracy and audio clarity being largely solved, sounding human and native is the differentiator.

    Every system reproduces the input text faithfully and produces clean audio. The real differences between systems are in how human and how native they sound, and the two are distinct: a voice can sound human without sounding native, and the reverse. Prosody matters most to evaluators.

    Takeaway

    Human-like and native-like delivery is the next frontier.

    The next gains are in pausing, prosody, native-sounding delivery and natural handling of numbers and code-switched content, especially in harder languages such as Malayalam. Targeted, dialect-specific native-speaker speech data is the fastest way to get there.

  3. 03
    Finding

    Rigorous evaluation design produces rankings that hold up and pinpoint core issues.

    Two independent methods, head-to-head comparison and one-at-a-time rating, produce the same overall ranking and closely matching per-language rankings. Separate questions for each property, plus breakdowns by language, domain and sentence type, isolate specific weak spots such as pausing, numbers and code-mixed speech.

    Takeaway

    Credible evaluation needs mixed methods and rigorous quality control.

    Combine comparative and absolute judgments, measure each quality separately, and set decision rules before the data comes in. Monitor rater reliability throughout, so that conclusions are shaped by strong and consistent data.

What makes MicCheck different

Text-to-speech (TTS) turns written text into spoken audio. It sits behind voice assistants, helplines, banking phone menus and agricultural advisory services. Several multilingual TTS leaderboards exist, yet there is still little independent evidence of how these voices sound to native speakers of Indian languages.

Tested where voice AI is used.

The most motivating use case for TTS in India is a voicebot: someone asks a question and listens to a spoken answer. We recreate that setting, taking real community questions, generating chatbot answers and testing each voice on a benchmark constructed from answers. Chatbots write differently from people, so voices need to be tested on the language chatbots actually produce.

A rubric built for native speakers.

Qualities like naturalness are hard to define, and even experts struggle to agree on them. We replaced them with six plain-language questions about what evaluators can actually hear and judge, and which translate easily across languages. Checks throughout make sure evaluators agree with themselves and with each other.

Which voice is best at what.

A useful evaluation shows which systems are best at which tasks, as well as which are best overall. Every system is tested on a balanced mix of sentence types, from code-mixed speech to numbers, with enough sentences in each category to give reliable results.

Results you can rely on.

Every score carries a confidence range, differences are corrected for the number of comparisons made, and two independent studies check each other. Where two systems cannot be separated, the leaderboard says so.

The test set stays private, so that no model can be trained on it or tuned to game it.

Two studies, one leaderboard

A Comparative study, in which evaluators pick the better of two voices, ranks the models. A Standalone study, which follows the framework, shows how good each voice is in absolute terms. The two studies use different tasks and different statistics, so where they agree on the order of systems, the ranking is much more trustworthy than either study alone.

Mean Opinion Score. A Mean Opinion Score (MOS) measures how good audio sounds to people. An evaluator hears one clip on its own and rates it on a scale of 1 to 5, and the score is the average of those ratings. In MicCheck, evaluators rate each clip on six dimensions of audio quality.

The benchmark

The benchmark is made of the kind of sentences people would see or hear from a voice assistant in India.

We took chatbot answers to real user questions across nine languages and four domains (Agriculture, Healthcare, Education and Finance), and carefully sampled a diverse selection of sentences from them for the benchmark. Native-speaker evaluators accepted 4,993 of the 5,400 sentences selected. The sentences are written the way a voice assistant would speak to someone in India: conversational, with English words where people naturally use them, numbers written as digits, and clarifying questions where needed.

The nine languages reflect India's diversity, from widely supported Hindi, through languages such as Bengali, Tamil, Telugu and Gujarati, to lower-resource languages like Odia. They span the Indo-Aryan and Dravidian language families and several scripts.

  • Bengali
  • Gujarati
  • Hindi
  • Indian English
  • Malayalam
  • Odia
  • Tamil
  • Telugu
  • Urdu

Sentence Selection Pipeline

  1. About 18,000 questions, mostly from real users

  2. Half answered by Gemma and half by Qwen, two open-weight models

  3. About 153,000 candidate sentences

  4. About 141,000 after automatic filters

  5. 5,400 selected, 150 per language per domain

  6. 4,993 accepted by native-speaker evaluators

  7. 3,387 used in the two listening studies

Sentence Selection Process

Stage 1 — A real user question

(Agriculture, English)

"I've heard about Nano Urea; is it really a complete replacement for bagged Urea, and how do I apply it on my paddy fields?"

Stage 2 — Answered by an open-weight model

Routed to Gemma

"Actually, Nano Urea isn't meant to completely replace bagged urea, but it's designed to reduce the amount of it you need. The general idea is that you can replace about 50 percent of your conventional urea bags with the liquid nano version. This helps cut down on costs and prevents too much nitrogen from leaching into the soil.

As for applying it to your paddy fields, you don't put it in the soil. Instead, you spray it directly on the leaves using a sprayer. You'll need to mix a small amount of Nano Urea with water, usually about 2 to 4 millilitres per litre, and spray it when the crop is at the right growth stage. It's best to do this during the foliage stage when the leaves are healthy.

Are you planning to use it as a standalone supplement or are you combining it with a basal dose of granular urea?"

Stage 3 — Sentences sampled into benchmark set

Sentence type
  1. 1"Actually, Nano Urea isn't meant to completely replace bagged urea, but it's designed to reduce the amount of it you need."ordinaryReserve
  2. 2"The general idea is that you can replace about 50 percent of your conventional urea bags with the liquid nano version."numeralsSelected
  3. 3"This helps cut down on costs and prevents too much nitrogen from leaching into the soil."ordinaryReserve
  4. 4"As for applying it to your paddy fields, you don't put it in the soil."ordinaryReserve
  5. 5"Instead, you spray it directly on the leaves using a sprayer."ordinaryReserve
  6. 6"You'll need to mix a small amount of Nano Urea with water, usually about 2 to 4 millilitres per litre, and spray it when the crop is at the right growth stage."longnumeralsSelected
  7. 7"It's best to do this during the foliage stage when the leaves are healthy."ordinaryReserve
  8. 8"Are you planning to use it as a standalone supplement or are you combining it with a basal dose of granular urea?"questionReserve

Why these two: they are the only sentences in the answer carrying numbers — "50 percent", "2 to 4 millilitres per litre". Everything else is a plain statement competing for the most crowded quota.

Where the sentences come from

  • Real questions first. The questions, mostly asked by real people, come from Karya's Samiksha project and previous studies.
  • Filling coverage gaps. Where a language or domain had too few real questions, we generated additional ones modelled on related languages, and native speakers checked samples of them.
  • Two models write the answers. Gemma and Qwen answered every question with identical settings, balanced roughly half each within every sentence type. Why: every model has its own habits, such as longer sentences, more English or more numbers, and with one model those habits would quietly shape the whole benchmark.
  • Open-weight models only. Both models are open-weight: their underlying weights are public and they can be run locally. Why: none of the benchmark text is produced by the commercial providers being evaluated.

How the sentences were selected

  • Automatic filters: removed sentences with problems of length, script or duplication.
  • Targeted audits: fixed subtler issues, such as how Urdu sentences were split and hidden traces of other scripts.
  • Balanced selection: chose 150 sentences per language per domain, with a minimum quota for each of the five sentence types.
  • Native-speaker review: rejected 7.5% of the selected sentences, from 1% in Indian English to about 20% in Tamil, catching problems no automatic check could find.
  • Reproducible: the same inputs always produce exactly the same benchmark.

Five sentence types

Each language and domain includes a minimum quota of five sentence types: plain everyday sentences, and four kinds where text-to-speech voices tend to stumble.

Five sentence types, with one example in Bengali
Sentence type Example (Bengali) English translation
Sentences that mix in words from another language, as people naturally do in everyday speech. আপনি ঠিক কোন পরীক্ষার জন্য প্রস্তুতি নিতে চাইছেন, যেমন JEE নাকি NEET, সেটা জানালে আমি আপনাকে আরও নির্দিষ্ট করে বলতে পারব। If you let me know exactly which exam you’re preparing for, such as JEE or NEET, I’ll be able to give you more specific guidance.
Sentences containing digits in any script, or patterns such as phone numbers, dates, ordinals like "3rd", ranges, prices, percentages and decimals. শুধু গরু কেনা, শেড তৈরি আর প্রাথমিক সেটআপের জন্য কয়েক লক্ষ থেকে শুরু করে ২০ থেকে ৩০ লক্ষ টাকা পর্যন্ত খরচ হতে পারে। The cost of purchasing cows, building the shed and the initial setup alone can range from a few lakhs to ₹20–30 lakh.
Sentences that end in a question mark (? or ؟), where the voice must carry question intonation. আপনি কি বিশেষ কোনো দেশের বর্তমান রিজার্ভ পরিস্থিতি নিয়ে জানতে চাইছেন, নাকি এটা সাধারণ ধারণা নেওয়ার জন্য জিজ্ঞেস করছেন? Are you asking about the current reserve situation of one particular country, or are you asking to get a general idea?
The longest sentences in each language, where pacing and pausing matter most. বিশ্ব স্বাস্থ্য সংস্থা বা ওয়ার্ল্ড হেলথ অর্গানাইজেশন বলে যে, যেহেতু পোলিও ভাইরাস এখনও বিশ্বের অন্য কিছু দেশে, বিশেষ করে পাকিস্তান এবং আফগানিস্তানে তামাশা করছে, তাই ভারতের মতো উন্নয়নশীল দেশেও রোগটি আবার ফিরে আসার ঝুঁকি সবসময়ই থাকে। According to the World Health Organization (WHO), since the polio virus is still circulating in some countries, particularly Pakistan and Afghanistan, there is always a risk of the disease returning even in a developing country like India.
Everyday sentences that fall outside the other four types, selected so that their sounds occur as often as they do in the language (phonetically balanced). একজন শিক্ষককে অনলাইনে ক্লাস নেওয়ার জন্য প্রথমেই কিছু বেসিক টেকনিক্যাল জিনিস গুছিয়ে নিতে হবে। To take classes online, a teacher first has to get some basic technical things in order.

Native Speaker Evaluation Methodology

Native-speaker evaluators judged every voice in two studies, both drawing on the benchmark, with checks at every stage so that the rankings reflect what users actually hear.

01. Comparative study

Evaluators hear the same sentence spoken by two systems, choose the one they prefer, and say why.

Clip 1
0:00 / 0:11
Clip 2
0:00 / 0:07
Which clip sounds better?
36model pairs compared per language when all nine systems are available
65sentences per language and domain, shared by all pairs
  • Input selection. Model inputs are spread evenly across sentence types and rotated across system pairs, so any two pairs share some inputs and no system gets easier text.
  • Reasons for each choice. After choosing, evaluators also said why, from a short set of reasons about how the voices sounded.
  • Why this study carries the ranking. Both clips use the same sentence and the same evaluator, so differences in sentence difficulty and evaluator strictness cancel out.

02. Standalone study

Follows the Mean Opinion Score (MOS) framework

Evaluators rate each clip on its own across six dimensions of voice quality, which shows how good each voice is in absolute terms.

0:00 / 0:15
Rate the clip on each of the six dimensions, from 1 to 5.
–
–
–
–
–
–
Clip scoreThe average of your six ratings, on a 1 to 5 scale

0 of 6 rated

–

03. How we made sure the numbers are reliable

Checks run before evaluators start, while they work and after the data is collected, and every comparison is set up to be fair.

Karya brings native-speaker evaluators at scale to voice AI, in languages most evaluations overlook. Every voice is tested on sentences drawn from how people actually use voice assistants, with rigorous checks behind every judgement. For MicCheck, around 600 evaluators took part, evenly split between men and women.

  1. Community evals: judge what works
  2. Collect: build the data
  3. Build models: train the core
  4. Build technology: deploy useful tools

Evaluate your voice AI with Karya

MicCheck is one study. The next can be yours. Karya closes the loop: every gap an evaluation finds becomes the brief for the next round of data collection, until your voice AI works for your languages and your users.

  • If you build voice models

    See where your model fails, then improve it with targeted data for fine-tuning.

  • If you deploy voice products

    Choose the voice that works best for your users and their languages.

  • If you shape how voice AI is governed

    Get evidence for policy, language by language and domain by domain.