← All runs

Evaluation run

audio run (incl LibriSpeech)

6 July 2026, 11:18 · eval-engine · 54c20aa · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

51%
Light, ships on by default
72/140 cases passed · 249 ms typical
40%
Polish, experimental
48/120 cases passed · 267 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  35 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

hfp-disf-selfcorrect-nameBluetoothneeds review
HeardSend it to John uh i mean jane Ohai Mijane. 50% misheard
Was saidsend it to john uh i mean jane
InstantSend it to John Ohai Mijane.
LightSend it to John Ohai Mijane. 160 msthe cleanup was rejected, so the original was kept
PolishSend it to John Ohai Mijane. 165 msthe cleanup was rejected, so the original was kept
Why?the result still contains “John”, which this case forbids; guard rejected; the microphone misheard 50% of the words
hfp-disf-selfcorrect-timeBluetoothneeds review
HeardLet's meet at three 3 no wait four 4 o'clock. 25% misheard
Was saidlet's meet at three no wait four o'clock
InstantLet's meet at 3 no wait 4 o'clock.
LightLet's meet at 3 no wait 4 o'clock. 222 ms
PolishLet's meet at 3 no wait 4 o'clock. 227 ms
Why?the microphone misheard 25% of the words
hfp-real-emailBluetoothclean
HeardHey Sarah just wanted to follow up on the proposal UM, do you have time to hop on a quick call tomorrow around three 3? 4% misheard
Was saidhey sarah just wanted to follow up on the proposal um do you have time to hop on a quick call tomorrow around three
InstantHey Sarah just wanted to follow up on the proposal UM, do you have time to hop on a quick call tomorrow around 3?
LightHey Sarah, just wanted to follow up on the proposal. UM, Do you have time to hop on a quick call tomorrow around 3? 550 ms
PolishHey Sarah, just wanted to follow up on the proposal. UM, Do you have time to hop on for a quick call tomorrow around 3? 530 ms
hfp-real-longBluetoothneeds review
HeardSo I talked to the team today and we decided UM, that we're we are going to push the launch to next week because the the testing isn't done and we want were to make sure everything is solid. 11% misheard
Was saidso i talked to the team today and we decided um that we're going to push the launch to next week because the the testing isn't done and we want to make sure everything is solid
InstantSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid.
LightSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid. 732 msthe cleanup was rejected, so the original was kept
PolishSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were need to make sure everything is solid. 906 ms
Why light?the result still contains “um”, which this case forbids; guard rejected
hfp-real-messageBluetoothclean
HeardRunning like ten 10 minutes late traffic is crazy be there as soon as I can. 7% misheard
Was saidrunning like ten minutes late traffic is crazy be there as soon as i can
InstantRunning like 10 minutes late traffic is crazy be there as soon as I can.
LightRunning like 10 minutes late, traffic is crazy, be there as soon as I can. 409 ms
PolishRunning like 10 minutes late. Traffic is crazy. I'll be there as soon as I can. 436 ms
ls-1272-128104-0000Real speechclean
Heardmister Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. 0% misheard
Was saidmister quilter is the apostle of the middle classes and we are glad to welcome his gospel
Instantmister Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel.
LightMister Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. 457 ms
ls-1272-128104-0001Real speechclean
HeardNor is mister Quilter's manner less interesting than his matter. 0% misheard
Was saidnor is mister quilter's manner less interesting than his matter
InstantNor is mister Quilter's manner less interesting than his matter.
LightNor is Mister Quilter's manner less interesting than his matter. 320 ms
ls-1272-128104-0002Real speechclean
HeardHe tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. 0% misheard
Was saidhe tells us that at this festive season of the year with christmas and roast beef looming before us similes drawn from eating and its results occur most readily to the mind
InstantHe tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind.
LightHe tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. 746 ms
ls-1272-128104-0003Real speechclean
HeardHe has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca. 0% misheard
Was saidhe has grave doubts whether sir frederick leighton's work is really greek after all and can discover in it but little of rocky ithaca
InstantHe has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca.
LightHe has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca. 674 ms
ls-1272-128104-0004Real speechclean
HeardLinnell's pictures are a sort of up guards and at em atom paintings, and Mason's exquisite idylls are as national as a jingo poem. mister birket Burkitt Foster's landscapes landscapepes smile at one much in the same way that mister Carker used to flash his teeth, and mister John Collier gives his sitter a cheerful slap on the back. before he says, like a shampooer in a Turkish bath, next man! 6% misheard
Was saidlinnell's pictures are a sort of up guards and at em paintings and mason's exquisite idylls are as national as a jingo poem mister birket foster's landscapes smile at one much in the same way that mister carker used to flash his teeth and mister john collier gives his sitter a cheerful slap on the back before he says like a shampooer in a turkish bath next man
InstantLinnell's pictures are a sort of up guards and atom paintings, and Mason's exquisite idylls are as national as a jingo poem. mister Burkitt Foster's landscapepes smile at one much in the same way that mister Carker used to flash his teeth, and mister John Collier gives his sitter a cheerful slap on the back. before he says, like a shampooer in a Turkish bath, next man!
LightLinnell's pictures are a sort of up guards and atom paintings, and Mason's exquisite idylls are as national as a jingo poem. Mister Burkitt Foster's landscapepes smile at one much in the same way that Mister Carker used to flash his teeth, and Mister John Collier gives his sitter a cheerful slap on the back. Before he says, like a shampooer in a Turkish bath, next man! 1805 ms