← All runs

Evaluation run

self-correction chains audio

6 July 2026, 13:56 · eval-engine · 0169abd · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

55%
Light, ships on by default
77/140 cases passed · 288 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  19 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

tts-real-emailSyntheticclean
HeardHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around three 3. 4% misheard
Was saidhey sarah just wanted to follow up on the proposal um do you have time to hop on a quick call tomorrow around three
InstantHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around 3.
LightHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on a quick call tomorrow around 3? 686 ms
PolishHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on a quick call tomorrow around 3? 547 ms
tts-real-longSyntheticneeds review
HeardSo I talked to the team today and we decided UM, that we're we are going to push the launch to next week because the the testing isn't done and we want were to make sure everything is solid. 8% misheard
Was saidso i talked to the team today and we decided um that we're going to push the launch to next week because the the testing isn't done and we want to make sure everything is solid
InstantSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were to make sure everything is solid.
LightSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were to make sure everything is solid. 757 msthe cleanup was rejected, so the original was kept
PolishSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were need to make sure everything is solid. 734 ms
Why light?the result still contains “um” and “the the”, which this case forbids
tts-real-messageSyntheticclean
HeardRunning like ten 10 minutes late traffic is crazy be there as soon as I can. 7% misheard
Was saidrunning like ten minutes late traffic is crazy be there as soon as i can
InstantRunning like 10 minutes late traffic is crazy be there as soon as I can.
LightRunning like 10 minutes late, traffic is crazy, be there as soon as I can. 421 ms
PolishRunning like 10 minutes late. Traffic is crazy. I'll be there as soon as I can. 424 ms
tts-real-noteSyntheticclean
HeardTodo for tomorrow finish the report call the bank and pick up groceries. 0% misheard
Was saidtodo for tomorrow finish the report call the bank and pick up groceries
InstantTodo for tomorrow finish the report call the bank and pick up groceries.
LightTodo for tomorrow: finish the report, call the bank, and pick up groceries. 386 ms
PolishTodo for tomorrow: finish the report, call the bank, and pick up groceries. 360 ms
tts-sens-passwordSyntheticneeds review
HeardMy Wi-Fi password is hunter two three four Hunter234. 75% misheard
Was saidmy wifi password is hunter two three four
InstantMy Wi-Fi password is Hunter234.
LightMy Wi-Fi password is Hunter234. 282 ms
PolishMy Wi-Fi password is Hunter234. 295 msthe cleanup was rejected, so the original was kept
Why?the microphone misheard 75% of the words
tts-vocab-acronymSyntheticneeds review
HeardOur a p i API is getting rate limited. 38% misheard
Was saidour a p i is getting rate limited
InstantOur API is getting rate limited.
LightOur API is getting rate limited. 221 ms
PolishOur API is getting being rate limited. 192 ms
Why?the microphone misheard 38% of the words
tts-vocab-namesSyntheticclean
HeardLoop in Raj and Priya on the nvidia Innovidya Integration. 11% misheard
Was saidloop in raj and priya on the nvidia integration
InstantLoop in Raj and Priya on the Innovidya Integration.
LightLoop in Raj and Priya on the Innovidya Integration. 347 ms
PolishLoop in Raj and Priya are looping on the Innovidya Integration. 311 ms
tts-vocab-preserveSyntheticclean
HeardWe deployed the Parakeet model to production. 14% misheard
Was saidwe deployed the parakeet model to production
InstantWe deployed Parakeet model to production.
LightWe deployed the Parakeet model to production. 347 ms
PolishWe deployed the Parakeet model to production. 247 ms
tts-vocab-ragSyntheticneeds review
Heardthe rag Drag pipeline retrieves documents before the model answers. 22% misheard
Was saidthe rag pipeline retrieves documents before the model answers
InstantDrag pipeline retrieves documents before the model answers.
LightDrag pipeline retrieves documents before the model answers. 254 ms
PolishThe drag pipeline retrieves documents before the model answers. 238 ms
Why?the microphone misheard 22% of the words
tts-vocab-techSyntheticclean
HeardWe deploy with Kubernetes and Terraform on AWS. 0% misheard
Was saidwe deploy with kubernetes and terraform on aws
InstantWe deploy with Kubernetes and Terraform on AWS.
LightWe deploy with Kubernetes and Terraform on AWS. 313 ms
PolishWe deploy with Kubernetes and Terraform on AWS. 341 ms