tts-num-no-oneSyntheticclean
HeardNo one showed up to the review. 0% misheard
Was saidno one showed up to the review
InstantNo one showed up to the review.
LightNo one showed up to the review. 227 ms
PolishNo one showed up to for the review. 189 ms
Evaluation run
faithfulness scoring cleanup audio
Did it hear the words right?
Did it clean up correctly and safely?
55%
Light, ships on by default
77/140 cases passed · 255 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words · 19 the cleanup itself slipped
Every case
Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.
140 cases match
tts-num-one-daySyntheticneeds review
HeardRemind me in one day to two check the logs blocks. 33% misheard
Was saidremind me in one day to check the logs
InstantRemind me in one day 2 check blocks.
LightRemind me in one day 2 to check blocks. 1066 ms
PolishRemind me in one day 2 from now to check the blocks. 878 ms
Why?the microphone misheard 33% of the words
tts-num-one-of-themSyntheticclean
HeardOne of them is already broken. 0% misheard
Was saidone of them is already broken
InstantOne of them is already broken.
LightOne of them is already broken. 186 ms
PolishOne of them is already broken. 176 ms
tts-num-one-on-oneSyntheticneeds review
HeardLet's do a one on one one-on-one tomorrow. 0% misheard
Was saidlet's do a one on one tomorrow
InstantLet's do a one-on-one tomorrow.
LightLet's do a one-on-one tomorrow. 245 ms
PolishLet's do a one-on-one tomorrow. 249 ms
Why?the result is missing “one on one”
tts-num-one-secSyntheticneeds review
HeardI'll be ready in one sec second. 17% misheard
Was saidi'll be ready in one sec
InstantI'll be ready in one second.
LightI'll be ready in one second. 227 ms
PolishI'll be ready in one second. 228 ms
Why?the microphone misheard 17% of the words
tts-num-percentSyntheticneeds review
HeardRevenue grew twenty five percent 25% last quarter. 43% misheard
Was saidrevenue grew twenty five percent last quarter
InstantRevenue grew 25% last quarter.
LightRevenue grew 25% last quarter. 211 ms
PolishRevenue grew 25% last quarter. 206 ms
Why?the microphone misheard 43% of the words
tts-num-phoneSyntheticneeds review
HeardCall me at five five five one two three four 5551234. 70% misheard
Was saidcall me at five five five one two three four
InstantCall me at 5551234.
LightCall me at 555-1234. 316 ms
PolishPlease call me at 555-1234. 333 ms
Why?the microphone misheard 70% of the words
tts-num-rangeSyntheticneeds review
HeardIt should take two to three 2-3 days. 43% misheard
Was saidit should take two to three days
InstantIt should take 2-3 days.
LightIt should take 2-3 days. 216 ms
PolishIt should take 2-3 2 to 3 days. 229 ms
Why?the microphone misheard 43% of the words
tts-num-roomSyntheticneeds review
HeardThe meeting is in room two oh five 205. 38% misheard
Was saidthe meeting is in room two oh five
InstantThe meeting is in room 205.
LightThe meeting is in room 205. 239 ms
PolishThe meeting is in room 205. 228 ms
Why?the microphone misheard 38% of the words
tts-num-room-214Syntheticneeds review
HeardThe interview is in room two fourteen 214. 29% misheard
Was saidthe interview is in room two fourteen
InstantThe interview is in room 214.
LightThe interview is in room 214. 265 ms
PolishThe interview is in room 214. 273 ms
Why?the microphone misheard 29% of the words