← All runs

Evaluation run

audio run (post ITN fix)

6 July 2026, 13:24 · eval-engine · 059e772 · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise17.3%

Did it clean up correctly and safely?

51%
Light, ships on by default
71/140 cases passed · 263 ms typical
41%
Polish, experimental
49/120 cases passed · 290 ms typical
When a case fails, whose fault is it?
103 the microphone misheard the words  ·  37 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

noisy-disf-false-startsNoisyneeds review
HeardSo the the main my thing is uh the main that map point is we need to ship shift. 47% misheard
Was saidso the the main thing is uh the main point is we need to ship
InstantSo my thing is that map point is we need to shift.
LightSo my thing is that map point is we need to shift. 408 ms
PolishSo my thing is that map point is we need to shift. 372 ms
Why?the result is missing “we need to ship”; the microphone misheard 47% of the words
noisy-disf-heavy-stutterNoisyclean
HeardI I I think we we should just go go now. 0% misheard
Was saidi i i think we we should just go go now
InstantI I I think we we should just go go now.
LightI I I think we we should just go go now. 269 ms
PolishI I I think we we should just go go now. 291 ms
noisy-disf-repeatNoisyclean
Heardthe Tha the report is is ready to go. 13% misheard
Was saidthe the report is is ready to go
InstantTha the report is is ready to go.
LightTha The report is is ready to go. 191 ms
PolishTha The report is is ready to go. 225 ms
noisy-disf-restartNoisyclean
HeardI think we should UM, I think we should just ship it. 0% misheard
Was saidi think we should um i think we should just ship it
InstantI think we should UM, I think we should just ship it.
LightI think we should UM, I think we should just ship it. 218 ms
PolishI think we should UM, I think we should just ship it. 258 ms
noisy-disf-scratchNoisyclean
HeardThe deadline is Friday actually no let's say next Monday. 0% misheard
Was saidthe deadline is friday actually no let's say next monday
InstantThe deadline is Friday actually no let's say next Monday.
LightThe deadline is Friday actually no let's say next Monday. 175 ms
PolishThe deadline is Friday actually no let's say next Monday. 186 ms
noisy-disf-selfcorrect-nameNoisyneeds review
HeardSend it to John uh i mean jane Uhai Mijane. 50% misheard
Was saidsend it to john uh i mean jane
InstantSend it to John Uhai Mijane.
LightSend it to John Uhai Mijane. 158 msthe cleanup was rejected, so the original was kept
PolishSend it to John Uhai Mijane. 172 msthe cleanup was rejected, so the original was kept
Why?the result still contains “John”, which this case forbids; guard rejected; the microphone misheard 50% of the words
noisy-disf-selfcorrect-timeNoisyneeds review
HeardLet's meet at three 3 no wait four 4 o'clock. 25% misheard
Was saidlet's meet at three no wait four o'clock
InstantLet's meet at 3 no wait 4 o'clock.
LightLet's meet at 3 no wait 4 o'clock. 220 ms
PolishLet's meet at 3 no wait 4 o'clock. 229 ms
Why?the microphone misheard 25% of the words
noisy-real-emailNoisyclean
HeardHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around three 3. 4% misheard
Was saidhey sarah just wanted to follow up on the proposal um do you have time to hop on a quick call tomorrow around three
InstantHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around 3.
LightHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on a quick call tomorrow around 3? 542 ms
PolishHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on for a quick call tomorrow around 3? 634 ms
noisy-real-longNoisyneeds review
HeardSo I talked to the team today and we decided UM, that we're we are going to push the launch to next week because the the testing isn't done and we want were to make sure everything is solid. 11% misheard
Was saidso i talked to the team today and we decided um that we're going to push the launch to next week because the the testing isn't done and we want to make sure everything is solid
InstantSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid.
LightSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid. 707 msthe cleanup was rejected, so the original was kept
PolishSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were want to make sure everything is solid. 833 ms
Why light?the result still contains “um”, which this case forbids; guard rejected
noisy-real-messageNoisyclean
HeardRunning like ten 10 minutes late traffic is crazy be there as soon as I can. 7% misheard
Was saidrunning like ten minutes late traffic is crazy be there as soon as i can
InstantRunning like 10 minutes late traffic is crazy be there as soon as I can.
LightRunning like 10 minutes late, traffic is crazy, be there as soon as I can. 388 ms
PolishRunning like 10 minutes late. Traffic is crazy. I'll be there as soon as I can. 528 ms