← All runs

Evaluation run

audio run (post ITN fix)

6 July 2026, 13:24 · eval-engine · 059e772 · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise17.3%

Did it clean up correctly and safely?

51%
Light, ships on by default
71/140 cases passed · 263 ms typical
41%
Polish, experimental
49/120 cases passed · 290 ms typical
When a case fails, whose fault is it?
103 the microphone misheard the words  ·  37 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

tts-real-emailSyntheticclean
HeardHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around three 3. 4% misheard
Was saidhey sarah just wanted to follow up on the proposal um do you have time to hop on a quick call tomorrow around three
InstantHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around 3.
LightHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on a quick call tomorrow around 3? 537 ms
PolishHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on for a quick call tomorrow around 3? 636 ms
tts-real-longSyntheticneeds review
HeardSo I talked to the team today and we decided UM, that we're we are going to push the launch to next week because the the testing isn't done and we want were to make sure everything is solid. 8% misheard
Was saidso i talked to the team today and we decided um that we're going to push the launch to next week because the the testing isn't done and we want to make sure everything is solid
InstantSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were to make sure everything is solid.
LightSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were to make sure everything is solid. 700 msthe cleanup was rejected, so the original was kept
PolishSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were want to make sure everything is solid. 889 ms
Why light?the result still contains “um” and “the the”, which this case forbids; guard rejected
tts-real-messageSyntheticclean
HeardRunning like ten 10 minutes late traffic is crazy be there as soon as I can. 7% misheard
Was saidrunning like ten minutes late traffic is crazy be there as soon as i can
InstantRunning like 10 minutes late traffic is crazy be there as soon as I can.
LightRunning like 10 minutes late, traffic is crazy, be there as soon as I can. 388 ms
PolishRunning like 10 minutes late. Traffic is crazy. I'll be there as soon as I can. 520 ms
tts-real-noteSyntheticclean
HeardTodo for tomorrow finish the report call the bank and pick up groceries. 0% misheard
Was saidtodo for tomorrow finish the report call the bank and pick up groceries
InstantTodo for tomorrow finish the report call the bank and pick up groceries.
LightTodo for tomorrow: finish the report, call the bank, and pick up groceries. 359 ms
PolishTodo for tomorrow: finish the report, call the bank, and pick up groceries. 440 ms
tts-sens-passwordSyntheticneeds review
HeardMy Wi-Fi password is hunter two three four Hunter234. 75% misheard
Was saidmy wifi password is hunter two three four
InstantMy Wi-Fi password is Hunter234.
LightMy Wi-Fi password is Hunter234. 264 ms
PolishMy Wi-Fi password is Hunter234. 290 msthe cleanup was rejected, so the original was kept
Why light?the microphone misheard 75% of the words
Why polish?guard rejected; the microphone misheard 75% of the words
tts-vocab-acronymSyntheticneeds review
HeardOur a p i API is getting rate limited. 38% misheard
Was saidour a p i is getting rate limited
InstantOur API is getting rate limited.
LightOur API is getting rate limited. 187 ms
PolishOur API is getting being rate limited. 233 ms
Why?the microphone misheard 38% of the words
tts-vocab-namesSyntheticclean
HeardLoop in Raj and Priya on the nvidia Innovidya Integration. 11% misheard
Was saidloop in raj and priya on the nvidia integration
InstantLoop in Raj and Priya on the Innovidya Integration.
LightLoop in Raj and Priya on the Innovidya Integration. 285 ms
PolishLoop in Raj and Priya are looping on the Innovidya Integration. 362 ms
tts-vocab-preserveSyntheticclean
HeardWe deployed the Parakeet model to production. 14% misheard
Was saidwe deployed the parakeet model to production
InstantWe deployed Parakeet model to production.
LightWe deployed the Parakeet model to production. 233 ms
PolishWe deployed the Parakeet model to production. 254 ms
tts-vocab-ragSyntheticneeds review
Heardthe rag Drag pipeline retrieves documents before the model answers. 22% misheard
Was saidthe rag pipeline retrieves documents before the model answers
InstantDrag pipeline retrieves documents before the model answers.
LightDrag pipeline retrieves documents before the model answers. 219 ms
PolishThe drag pipeline retrieves documents before the model answers. 290 ms
Why?the microphone misheard 22% of the words
tts-vocab-techSyntheticclean
HeardWe deploy with Kubernetes and Terraform on AWS. 0% misheard
Was saidwe deploy with kubernetes and terraform on aws
InstantWe deploy with Kubernetes and Terraform on AWS.
LightWe deploy with Kubernetes and Terraform on AWS. 317 ms
PolishWe deploy with Kubernetes and Terraform on AWS. 305 ms