Over ten refinement steps, the semantic error rate on the English-language GigaSpeech benchmark falls from 21.47% to 3.49%. Word Error Rate on the same set, over the same ten steps, moves from 11.92% to 10.43%.
One transcript, one repair, two metrics. The one used to choose a recognition provider barely noticed.
The two columns sit side by side in Table II of the Agentic ASR paper (arXiv:2605.29430), a preprint submitted on 28 May 2026. The authors are from the X-LANCE Lab at Shanghai Jiao Tong University, Fudan University, the Chinese University of Hong Kong (Shenzhen) and the Tongyi Fun team at Alibaba Group — which explains the choice of models: in the experiment, both recognition and correction are done by Qwen models. The way this paper repairs a transcript also explains the gap between the columns.
Why "fix the transcript" breaks as a command
The typical setup on top of recognition looks like this: ASR produces a hypothesis, and a language model cleans it up. The setup holds for exactly as long as the user silently accepts the result.
The paper opens with a scene anyone who has talked to a voice assistant will recognize. A person asks: "Ask Megan to send me the budget file." The system hears Morgan and cheerfully replies that it will ask Morgan. The person objects: "No! It's Megan." That utterance goes through recognition too, and to the system it is no different from the previous one — just another chunk of speech to transcribe and append. The correction lands in the transcript instead of changing it. Fig. 1 of the paper lays the scene out in three columns, and the middle one — the traditional setup — ends tellingly: the person loses patience and spells the name out, "No! It's Megan, M-e-g-a-n," and the system takes exactly that line and writes it down as recognized text.
Before anything can be edited, the system has to learn to recognize that what it just heard was an edit. In Agentic ASR this is a separate step — intent routing, which assigns the utterance to one of three categories:
Intent | What the system does |
|---|---|
| leaves the current transcript state unchanged |
| appends the utterance as new content |
| edits the existing transcript state |
Only the third branch leads to the editor; the other two never invoke it. Without this step, every "no, that's wrong" becomes part of the text.
Three operations instead of one rewrite
Once the intent is identified as a correction, the editor kicks in. The paper writes the formula as a composition of three operations: Modify ∘ Reason ∘ Locate.
Locate finds the fragment that needs to change. Reason infers from the utterance and the conversation context what exactly the edit should be. Modify applies it at that spot.
The authors state the choice plainly: instead of an unconstrained one-step rewrite, correction is decomposed into three explicit operations. The motivation given in the text is a more controllable process and a better match with how people give partial repair instructions. Nobody dictates the whole sentence again when a single name needs to change.
The difference between the two approaches is sharpest on proper names. Rewriting is generation: the model reproduces the whole utterance, including the words that were recognized correctly. On every repeat pass, a rare surname, a village name or a product SKU can slip toward a more frequent neighboring form, and that is exactly how Megan turns into Morgan. Editing a fragment leaves no room for that: whatever was not touched stays as it was.
For Ukrainian this is not an abstraction. Surnames in the vocative case, names of towns and villages outside the regional capitals, brand names in Latin script inside a Ukrainian utterance — this is exactly the class of words where rewriting costs the most.
The metric that doesn't see the repair
Along with the framework, the authors introduce a metric of their own — the Sentence-level Semantic Error Rate, S²ER. It measures the share of utterances that failed to preserve the intended meaning: a judge model gives a binary answer, "meaning preserved / not preserved," looking at the main intent and the key meaning-bearing content, proper names above all, and deliberately ignoring filler words, disfluencies and punctuation.
On the English-language GigaSpeech Test, this is how the two metrics behave as refinement steps are added.
Refinement step | S²ER (meaning) | WER (words) |
|---|---|---|
0 — initial hypothesis | 21.47% | 11.92% |
1 | 12.35% | 11.02% |
3 | 7.00% | 10.69% |
10 | 3.49% | 10.43% |
Over ten steps, the semantic error rate shrinks sixfold. WER — the share of incorrectly recognized words — loses a point and a half over the same span. The authors put their conclusion dryly: on their own, surface-form metrics substantially understate the practical value of interaction.
So that S²ER would not have to be taken on trust, the authors checked the judge against people: 120 utterances from three language conditions, 25 non-experts and 5 specialists, a binary protocol. The Pearson correlation between the judge model and the averaged human score is 0.891 on GigaSpeech, 0.828 on WenetSpeech and 0.903 on ASRU2019, and on all three sets it is slightly higher than the correlation of the experts themselves with that same averaged score.
Two utterances that show where WER gets it wrong
The gap between the columns is best explained by a pair of examples from the paper itself.
What was said | What was recognized | WER | Meaning |
|---|---|---|---|
"Um, let's maybe just open the window?" | "Let's open the window?" | 42.9% | preserved |
"Try Qwen3-ASR to get the transcript!" | "Try Kunthreesir to get the transcript!" | 16.7% | broken |
In the first row almost half the words did not match — and none of them mattered, because what differed was fillers and softeners. The utterance can still be acted on in full. In the second row there is a single error and WER is 2.6 times lower, yet the command can no longer be carried out: the swapped model name is exactly what the next action depends on.
In a voice agent that qualifies leads, the same mechanism plays out on place names. A transcript that dropped a "ну" ("well") and a "власне" ("actually") costs nothing. A transcript in which Kropyvnytskyi became Kremenchuk sends the lead to the wrong stage — and from the outside it looks like the system working flawlessly. We covered the logic of decision fields, the ones that determine the route, in our piece on where a voice agent breaks in Ukrainian; Agentic ASR gives the same idea a formal metric and proves it with numbers.
What carries over to Ukrainian and what doesn't
The measurements were made on English (GigaSpeech), Mandarin (WenetSpeech, AISHELL-NER) and mixed English-Mandarin speech (ASRU2019, CS-Dialogue). We found no such study for Ukrainian — we checked arXiv and the authors' own publications. The percentages cannot be carried over to Ukrainian, but the structure of the results is worth reading.
Condition | S²ER at the start | S²ER after 10 steps |
|---|---|---|
Named entities (AISHELL-NER Test) | 19.91% | 2.02% |
Mixed speech, read utterances (ASRU2019) | 28.57% | 1.36% |
Mixed speech, spontaneous conversation (CS-Dialogue) | 19.73% | 4.16% |
The top two rows are exactly the classes that break Ukrainian telephony: proper names and switching between languages. Both can be repaired.
CS-Dialogue, in the bottom row, is spontaneous conversational language switching, the closest thing in the whole set to a real call, and it is the hardest to correct: 4.16% against 1.36% in the neighboring condition. The token-level metric there barely moves at all: 14.44% at the start against 13.28% after ten steps. The authors attribute this to the greater difficulty of spontaneous conversational code-switching.
Surzhyk, the everyday mix of Ukrainian and Russian, is on a phone call just that: spontaneous language switching within an utterance. The closest available analogue is the hardest case, and it is that behavior you should plan for, not the best row in the table.
The limit of the result: the user in this experiment is a model too
The ten refinement steps are not performed by a human, and in retellings of the paper this detail is the first to vanish. The authors built an Interactive Simulation System: the Qwen3-32B model compares the current transcript with the ground truth and formulates a correction, and Index-TTS-1.5 voices it in the same speaker's voice. That same Qwen3-32B serves as three different components — the correction model, the correction generator in the user simulator, and the semantic judge.
In other words, the other party in the experiment sees the right answer, is always available and never tires of correcting ten times in a row. None of those three properties survives contact with a live caller phoning a company.
The practical takeaway comes from the authors themselves, and it holds up better than the final figures: most of the semantic benefit arrives early. On GigaSpeech the very first refinement takes the error rate from 21.47% to 12.35% — roughly 42% of errors with a single re-ask. On ASRU2019 one step gives 28.57% → 10.32%, or about 64%. In a production scenario the tenth step will never come, and the first one always will.
What this changes in the price and timeline of a pilot
For a business owner, three practical consequences follow from this paper, and all three affect the cost estimate.
First: recognition testing moves into the pilot. There is no public number for Ukrainian telephony, so the only usable number is the one obtained on your own recordings — a few dozen real calls, labeled ground truth, a run of several candidates on the same set. That is a few days of work inside the pilot, not a separate research phase before it.
Second: the acceptance threshold is fixed not in WER but in accuracy on the decision fields and the share of utterances where the meaning was preserved. The threshold number is agreed before the start and becomes the very KPI the pilot is judged by. A clause like "recognition accuracy no worse than N%" does not work in a contract: it measures something the lead's route does not depend on.
Third: a voice scenario is not the cheapest pilot. Voice and multilingual support are part of the Operator plan: from €6,000 for a 30-day pilot, then from €149 a month plus tokens; text channels start with Starter at €1,800 for 14 days and from €49 a month. The pilot is paid up front; if the agreed KPI has not moved at the day-30 review, we refund it in full. So when the same process can be tested in chat, it is cheaper to start with chat and add voice once the qualification logic has been confirmed by numbers.
What to measure instead of WER
A voice agent's acceptance metric has to answer the question "can what was heard be acted on?" rather than "how many words matched?"
The working minimum is three measurements on your own recordings. The first is the share of utterances in which the meaning was not preserved: a binary judgment for each utterance, on the same principle as S²ER. The second is accuracy on the fields the lead's route depends on, separately from the rest of the transcript. The third is the re-ask rate and the share of conversations where the caller had to correct the same thing twice; this metric catches what neither WER nor S²ER shows — caller fatigue.
The third measurement is the cheapest and almost always last in line, if anyone remembers it at all. Yet it is the first to show the moment an agent has learned to repair the transcript at the expense of the person on the other end of the line.
For a business owner, everything above comes down to one line in the cost estimate: recognition testing lives in the pilot, not in procurement, and the acceptance threshold is written in decision fields, not in WER. If your contractor is ready to put that in the contract, you can stop reading here. Below is what an engineer takes from this paper.
For the technical team: what this means for the architecture
Three sentences from the paper translate into code almost word for word. Correction is a separate step with its own contract, not a line in a big prompt: an instruction like "fix the transcript if there is an error in it" inside a general prompt gives you no intent routing, no fragment localization and no control over what the model will touch. Utterance classification comes before any work on the text; otherwise the caller's objection is added to the transcript instead of correcting it. The editor receives a location, not a sentence: the model returns a replacement for the located fragment, and the rest of the text stays untouched.
The fourth decision is harder, because it contradicts the first impression the table gives. After the row "10 steps, S²ER 3.49%" it is tempting to build a refinement loop into the script that keeps spinning until the agent is sure it has understood. On a live call that loop kills the conversation: a person asked for the name of their city a third time hangs up, and it will go down in the logs as a technically flawless session. There is one re-ask in the budget, and it should be spent right after the critical field has been spoken, while the context is still fresh, not at the end of the call, on one combined confirmation of all the fields.
How this fits into the overall launch plan is covered in the step-by-step checklist for deploying in two weeks and in the executive's guide to an agent on inbound leads. A correction layer does not remove the need to control the agent's actions — we covered that separately, in the piece on the three protection levels.
Prepared by Andrew Maryasov, founder of Grow2.ai, a company that deploys AI agents in business processes. The figures were checked against the arXiv:2605.29430 preprint as of 26 August 2026. Need a voice scenario assessed for your calls? Get in touch.