← All runs

Evaluation run

self-correction chains audio

6 July 2026, 13:56 · eval-engine · 0169abd · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

55%
Light, ships on by default
77/140 cases passed · 288 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  19 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

tts-grammar-numbers-keep-01Syntheticneeds review
HeardThe budget is like fifty thousand 50,000 no wait seventy five thousand save 85,000 for Q3. 38% misheard
Was saidthe budget is like fifty thousand no wait seventy five thousand for q3
InstantThe budget is like 50,000 no wait save 85,000 for Q3.
LightThe budget is like 50,000 no wait save 85,000 for Q3. 469 ms
PolishThe budget is like 50,000 no wait save $85,000 for Q3, not $50,000. 684 ms
Why light?the result is missing “75”; the microphone misheard 38% of the words
Why polish?the result is missing “75”; the result still contains “50,000” and “$50”, which this case forbids; the microphone misheard 38% of the words
tts-grammar-runon-01Syntheticclean
HeardSo I went to the meeting and then I talked to Sarah and then we decided to push the launch and also we need to tell the team. 0% misheard
Was saidso i went to the meeting and then i talked to sarah and then we decided to push the launch and also we need to tell the team
InstantSo I went to the meeting and then I talked to Sarah and then we decided to push the launch and also we need to tell the team.
LightSo I went to the meeting and then I talked to Sarah and then we decided to push the launch, and also we need to tell the team. 870 ms
PolishSo I went to the meeting and then I talked to Sarah. and then We decided to push the launch and also we need to tell the team. 732 ms
tts-grammar-tense-01Syntheticclean
HeardYesterday I go to the office and ain't I see the new design. 8% misheard
Was saidyesterday i go to the office and i see the new design
InstantYesterday I go to the office ain't I see the new design.
LightYesterday I go to the office ain't I see the new design. 429 msthe cleanup was rejected, so the original was kept
PolishYesterday I go went to the office ain't I and didn't see the new design. 360 ms
tts-num-chapterSyntheticneeds review
HeardLet's start with chapter one 1. 20% misheard
Was saidlet's start with chapter one
InstantLet's start with chapter 1.
LightLet's start with chapter 1. 214 ms
PolishLet's start with chapter 1. 224 ms
Why?the microphone misheard 20% of the words
tts-num-currencySyntheticneeds review
HeardThe plan costs twenty five dollars $25 a month. 38% misheard
Was saidthe plan costs twenty five dollars a month
InstantThe plan costs $25 a month.
LightThe plan costs $25 a month. 258 ms
PolishThe plan costs $25 a month. 207 ms
Why?the microphone misheard 38% of the words
tts-num-decimalSyntheticneeds review
HeardThe download is three point five gigabytes 3.5GB. 57% misheard
Was saidthe download is three point five gigabytes
InstantThe download is 3.5GB.
LightThe download is 3.5GB. 259 ms
PolishThe download is 3.5GB. 218 msthe cleanup was rejected, so the original was kept
Why?the microphone misheard 57% of the words
tts-num-decimal-pctSyntheticneeds review
HeardWe hit ninety nine point nine percent 99.9% uptime. 63% misheard
Was saidwe hit ninety nine point nine percent uptime
InstantWe hit 99.9% uptime.
LightWe hit 99.9% uptime. 275 ms
PolishWe hit 99.9% uptime. 284 ms
Why?the microphone misheard 63% of the words
tts-num-first-placeSyntheticclean
HeardShe came in first place at the meet. 0% misheard
Was saidshe came in first place at the meet
InstantShe came in first place at the meet.
LightShe came in first place at the meet. 238 ms
PolishShe came in first place at the meet. 219 ms
tts-num-halfSyntheticneeds review
HeardGive a me half a second to think sacred toothink. 57% misheard
Was saidgive me half a second to think
InstantGive a me half a sacred toothink.
LightGive a me half a sacred toothink. 213 msthe cleanup was rejected, so the original was kept
PolishGive a me half a sacred toothink. 305 msthe cleanup was rejected, so the original was kept
Why?the result is missing “half a second”; the microphone misheard 57% of the words
tts-num-largeSyntheticneeds review
HeardWe shipped one hundred and twenty 120 units. 57% misheard
Was saidwe shipped one hundred and twenty units
InstantWe shipped 120 units.
LightWe shipped 120 units. 211 ms
PolishWe shipped 120 units. 194 ms
Why?the microphone misheard 57% of the words