← All runs

Evaluation run

faithfulness scoring cleanup audio

6 July 2026, 13:38 · eval-engine · 232478e · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

55%
Light, ships on by default
77/140 cases passed · 255 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  19 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

hfp-corr-money-noBluetoothneeds review
HeardThe plan is twenty five 35 no forty dollars $40 a month. 40% misheard
Was saidthe plan is twenty five no forty dollars a month
InstantThe plan is 35 no $40 a month.
LightThe plan is 35 no $40 a month. 243 ms
PolishThe plan is $35 no $40 a month, not $40. 334 ms
Why?the microphone misheard 40% of the words
hfp-corr-name-chainBluetoothneeds review
HeardCall John no Jane, no, actually, Mike. 0% misheard
Was saidcall john no jane no actually mike
InstantCall John no Jane, no, actually, Mike.
LightCall John, no Jane, no, actually, Mike. 292 ms
PolishCall John, no not Jane. no, It's actually Mike. 289 ms
Why?the result still contains “John” and “Jane”, which this case forbids
hfp-corr-num-chainBluetoothneeds review
HeardWe need twenty 20, no thirty 30, no forty 40 units. 38% misheard
Was saidwe need twenty no thirty no forty units
InstantWe need 20, no 30, no 40 units.
LightWe need 20, no 30, no 40 units. 374 ms
PolishWe need 20, no 30, no 40 units. 384 ms
Why?the result still contains “20” and “30”, which this case forbids; the microphone misheard 38% of the words
hfp-corr-time-noBluetoothneeds review
HeardLet's meet at three 3 no four thirty 4.30. 43% misheard
Was saidlet's meet at three no four thirty
InstantLet's meet at 3 no 4.30.
LightLet's meet at 3 no 4:30. 258 ms
PolishLet's meet at 3 no 4:30. 265 ms
Why?the microphone misheard 43% of the words
hfp-disf-dayBluetoothclean
HeardLet's do it on Monday, sorry I mean Tuesday. 0% misheard
Was saidlet's do it on monday sorry i mean tuesday
InstantLet's do it on Monday, sorry I mean Tuesday.
LightLet's do it on Monday, sorry I mean Tuesday. 197 ms
PolishLet's do it on Monday, sorry I mean Tuesday. 198 ms
hfp-disf-false-startsBluetoothneeds review
HeardSo the the main thing is uh the main that map point is we need to ship. 27% misheard
Was saidso the the main thing is uh the main point is we need to ship
InstantSo the main thing is that map point is we need to ship.
LightSo the main thing is that map point is we need to ship. 336 ms
PolishSo the main thing is that map point is we need to ship. 288 ms
Why?the microphone misheard 27% of the words
hfp-disf-heavy-stutterBluetoothclean
HeardI I I think we we should just go go now. 0% misheard
Was saidi i i think we we should just go go now
InstantI I I think we we should just go go now.
LightI I I think we we should just go go now. 236 ms
PolishI I I think we we should just go go now. 255 ms
hfp-disf-repeatBluetoothclean
HeardThe the report is is ready to go. 0% misheard
Was saidthe the report is is ready to go
InstantThe the report is is ready to go.
LightThe the report is is ready to go. 186 ms
PolishThe the report is is ready to go. 194 ms
hfp-disf-restartBluetoothclean
HeardI think we should UM, I think we should just ship it. 0% misheard
Was saidi think we should um i think we should just ship it
InstantI think we should UM, I think we should just ship it.
LightI think we should UM, I think we should just ship it. 214 ms
PolishI think we should UM, I think we should just ship it. 221 ms
hfp-disf-scratchBluetoothclean
HeardThe deadline is Friday actually no, let's say next Monday. 0% misheard
Was saidthe deadline is friday actually no let's say next monday
InstantThe deadline is Friday actually no, let's say next Monday.
LightThe deadline is Friday actually no, let's say next Monday. 183 ms
PolishThe deadline is Friday actually no, let's say next Monday. 184 ms