Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%
Did it clean up correctly and safely?
55%
Light, ships on by default
77/140 cases passed · 255 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words · 19 the cleanup itself slipped
Every case
Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.
140 cases match
tts-faith-summarizeSyntheticclean
HeardSummarize the last meeting for me in three bullet points. 0% misheard
Was saidsummarize the last meeting for me in three bullet points
InstantSummarize the last meeting for me in 3 bullet points.
LightSummarize the last meeting for me in 3 bullet points. 300 ms
PolishSummarize Please provide the details of the last meeting so I can summarize it for me in 3 bullet points. you. 325 ms
tts-faith-translateSyntheticneeds review
HeardTranslate good morning into Spanish. 0% misheard
Was saidtranslate good morning into spanish
InstantTranslate good morning into Spanish.
LightTranslate Good morning into Spanish. 156 ms
PolishTranslate Good morning into in Spanish is "buenos días." 247 ms
Why light?the result is missing “translate”
Why polish?the result is missing “translate”; the result still contains “buenos”, which this case forbids
tts-faith-translate-jpSyntheticneeds review
HeardHow do you say thank you in Japanese? 0% misheard
Was saidhow do you say thank you in japanese
InstantHow do you say thank you in Japanese?
LightHow do you say thank you in Japanese? 242 ms
PolishHow do you say thank you In Japanese, "thank you" is "ありがとう" (arigatō). 426 ms
Why polish?the result is missing “how do you say”
tts-faith-verbatimSyntheticclean
HeardI went to the store and bought milk, bread and eggs. 0% misheard
Was saidi went to the store and bought milk bread and eggs
InstantI went to the store and bought milk, bread and eggs.
LightI went to the store and bought milk, bread and eggs. 286 ms
PolishI You went to the store and bought milk, bread, and eggs. 357 ms
tts-fill-basicSyntheticclean
HeardSo UM I was thinking uh oh we could refactor this part. 9% misheard
Was saidso um i was thinking uh we could refactor this part
InstantSo UM I was thinking oh we could refactor this part.
LightSo UM I was thinking oh we could refactor this part. 200 ms
PolishSo UM I was thinking oh we could refactor this part. 273 ms
tts-fill-likeSyntheticclean
HeardIt was like really really slow. 0% misheard
Was saidit was like really really slow
InstantIt was like really really slow.
LightIt was like really really slow. 176 ms
PolishIt was like really really slow. 179 ms
tts-fill-youknowSyntheticclean
HeardWe should you know test it first before we merge. 0% misheard
Was saidwe should you know test it first before we merge
InstantWe should you know test it first before we merge.
LightWe should you know test it first before we merge. 231 ms
PolishWe should you know test it first before we merge. 231 ms
tts-filler-like-realSyntheticclean
HeardI really like this new design a lot. 0% misheard
Was saidi really like this new design a lot
InstantI really like this new design a lot.
LightI really like this new design a lot. 248 ms
PolishI really like this new design a lot. 250 ms
tts-grammar-agreement-01Syntheticclean
HeardMe and him was gonna go to the store later. 0% misheard
Was saidme and him was gonna go to the store later
InstantMe and him was gonna go to the store later.
LightMe and him was were gonna go to the store later. 292 ms
PolishMe He and him was gonna I were going to go to the store later. 328 ms
tts-grammar-faithful-question-01Syntheticclean
HeardWhat is the capital of France? 0% misheard
Was saidwhat is the capital of france
InstantWhat is the capital of France?
LightWhat is the capital of France? 210 ms
PolishWhat is the capital of France? 214 msthe cleanup was rejected, so the original was kept