Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%
Did it clean up correctly and safely?
55%
Light, ships on by default
77/140 cases passed · 288 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words · 19 the cleanup itself slipped
Every case
Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.
140 cases match
tts-faith-summarizeSyntheticclean
HeardSummarize the last meeting for me in three bullet points. 0% misheard
Was saidsummarize the last meeting for me in three bullet points
InstantSummarize the last meeting for me in 3 bullet points.
LightSummarize the last meeting for me in 3 bullet points. 312 ms
PolishSummarize Please provide the details of the last meeting for me to summarize it in 3 bullet points. 396 ms
tts-faith-translateSyntheticneeds review
HeardTranslate good morning into Spanish. 0% misheard
Was saidtranslate good morning into spanish
InstantTranslate good morning into Spanish.
LightTranslate Good morning into Spanish. 168 ms
PolishTranslate Good morning into in Spanish is "buenos días." 254 ms
Why light?the result is missing “translate”
Why polish?the result is missing “translate”; the result still contains “buenos”, which this case forbids
tts-faith-translate-jpSyntheticclean
HeardHow do you say thank you in Japanese? 0% misheard
Was saidhow do you say thank you in japanese
InstantHow do you say thank you in Japanese?
LightHow do you say thank you in Japanese? 309 ms
PolishHow do you say "thank you" in Japanese? The word is "ありがとう" (arigatō). 465 ms
tts-faith-verbatimSyntheticclean
HeardI went to the store and bought milk, bread and eggs. 0% misheard
Was saidi went to the store and bought milk bread and eggs
InstantI went to the store and bought milk, bread and eggs.
LightI went to the store and bought milk, bread and eggs. 370 ms
PolishI You went to the store and bought milk, bread, and eggs. 362 ms
tts-fill-basicSyntheticclean
HeardSo UM I was thinking uh oh we could refactor this part. 9% misheard
Was saidso um i was thinking uh we could refactor this part
InstantSo UM I was thinking oh we could refactor this part.
LightSo UM I was thinking oh we could refactor this part. 258 ms
PolishSo UM I was thinking oh we could refactor this part. 251 ms
tts-fill-likeSyntheticclean
HeardIt was like really really slow. 0% misheard
Was saidit was like really really slow
InstantIt was like really really slow.
LightIt was like really really slow. 223 ms
PolishIt was like really really slow. 180 ms
tts-fill-youknowSyntheticclean
HeardWe should you know test it first before we merge. 0% misheard
Was saidwe should you know test it first before we merge
InstantWe should you know test it first before we merge.
LightWe should you know test it first before we merge. 257 ms
PolishWe should you know test it first before we merge. 235 ms
tts-filler-like-realSyntheticclean
HeardI really like this new design a lot. 0% misheard
Was saidi really like this new design a lot
InstantI really like this new design a lot.
LightI really like this new design a lot. 319 ms
PolishI really like this new design a lot. 315 ms
tts-grammar-agreement-01Syntheticclean
HeardMe and him was gonna go to the store later. 0% misheard
Was saidme and him was gonna go to the store later
InstantMe and him was gonna go to the store later.
LightMe and him was were gonna go to the store later. 337 ms
PolishMe He and him was gonna I were going to go to the store later. 384 ms
tts-grammar-faithful-question-01Syntheticclean
HeardWhat is the capital of France? 0% misheard
Was saidwhat is the capital of france
InstantWhat is the capital of France?
LightWhat is the capital of France? 240 ms
PolishWhat is the capital of France? 301 msthe cleanup was rejected, so the original was kept