A user sent us a screenshot last week. Their meeting transcript showed the same sentence twice, back to back, attributed to two different people:
Speaker 1
to stay to make a difference. It was part of the
first ever one city day.
Speaker 2
to stay to make a difference. It was part of the
first ever one city day.
Both copies got translated. Both counted against their credits. And if you looked closely, the copies were not quite identical - one had heard "Beerba Buena Gardens", the other "A Yerba Buena Gardens". Same audio, same moment, two transcriptions that disagreed about what was said. That last detail turned out to be the whole story.
Where the second copy came from
Foxl Notes transcribes meetings through Amazon Transcribe, streamed over a WebSocket the foxl.ai relay presigns. Until v0.4.0, live-translate mode captured exactly one audio source: the meeting's system audio. That had an obvious gap - your own questions never got translated, because your voice does not come out of your own speakers. So in v0.4.0 we did the obvious thing. Capture the microphone and the system audio, run a transcription pipeline for each, tag one set of segments "Speaker N" and the other "Remote N", translate everything.
On headphones this works fine. The mic hears you, the system-audio capture hears them, clean separation. On speakers - which is how a lot of people actually sit through meetings - the mic hears you and the room, and the room includes your speakers playing the other person's voice. Every remote utterance now reaches Amazon Transcribe twice: once through the digital capture, once through the air, a desk, and a laptop microphone. Two sockets, two transcriptions, two bills. Diarization has no idea these are the same person; acoustically they barely are.
The filter that almost worked
Our first fix, in v0.4.2, was the one most codebases reach for: detect the duplicates and drop one. Normalize both texts, tokenize, and if a finalizing mic segment shares at least 80% of its tokens with a nearby system-audio segment - nearby meaning within a 12-second window - treat it as speaker echo and delete it.
It worked on our desks. It survived our test recordings. And the very first real-world screenshot we got back showed it failing, and showed exactly why it had to fail. Look at the pair again: "Beerba Buena Gardens" versus "A Yerba Buena Gardens". The two acoustic paths are not copies of each other. One went through a speaker cone, three feet of air, and echo cancellation before hitting a laptop mic; the ASR model heard genuinely different audio and produced genuinely different words. Token overlap collapses precisely on the sentences where the two paths disagree most - proper nouns, fast speech, crosstalk - which are precisely the sentences where transcription is hardest. The filter was calibrated on the easy case and deployed against the hard one.
The 12-second window had the same problem in miniature. The two pipelines kept independent clocks - each one's timestamps anchored to its own socket's first audio frame - so the "same moment" in pipeline A and pipeline B could drift apart, and a duplicate arriving outside the window sailed straight through. Every constant in that filter was a knob, and every knob was a place to be wrong.
A threshold in a bugfix is a confession that you have not found the bug. You have found a price for it.
The fix was a delete
The version of Foxl Notes that never had this bug - everything before v0.4.0 - was not smarter than the code that followed it. It was structurally incapable of the bug, because there was only one stream. One stream cannot duplicate itself. In v0.4.0 we traded that impossibility for a probability, and in v0.4.2 we tried to buy the impossibility back with string comparison. It does not work like that.
v0.4.3 goes back to one stream without giving up the feature. The Web Audio API makes this almost embarrassingly easy: connect two MediaStreamSource nodes - the mic and the system audio - to the same processor node, and the browser sums the samples for you. One mixed signal, one WebSocket, one transcription. An utterance physically cannot reach Amazon Transcribe twice, because there is only one wire for it to travel down.
const sources = streams.map(s => ctx.createMediaStreamSource(s));
const processor = ctx.createScriptProcessor(4096, 1, 1);
sources.forEach(s => s.connect(processor)); // WebAudio sums them
Echo cancellation on the mic still matters - it strips the speaker playback out of the mic leg, so the mix carries the remote voice once, from the clean digital path, instead of once cleanly and once through the room. But it is now an audio-quality concern, not a correctness one. If AEC has a bad day, you get a slightly muddier signal, not a duplicated transcript.
The knock-on effects were all deletions. The similarity filter: gone. The "Remote N" label space and its collision-avoidance logic: gone - one stream means one diarization space, and Amazon's speaker separation does the job we were doing by hand. The saved audio recording, which had been mic-only and lost the remote party entirely on headphones, now records the same mix the transcriber hears. The commit that fixed a bug two releases of code had failed to fix removed more lines than it added.
What we are taking from it
None of this is a new lesson, which is exactly what makes it worth writing down. When a bug appears after a feature ships, the instinct is to keep the feature's shape and add a defense - a filter, a validator, a reconciliation pass. Sometimes that is right. But defenses inherit the failure modes of the thing they defend, plus their own. A dedup filter for two ASR streams has to be at least as good at deciding "same utterance" as the ASR is at transcribing - and if the ASR were that reliable, the two streams would agree and the filter would be trivial. We were asking the patch to be smarter than the system it patched.
The better question, the one we should have asked in the v0.4.2 planning five minutes in, is: what would make this failure impossible instead of handled? Sometimes the answer is expensive. This time it was three lines of Web Audio and the confidence to delete our own week-old code.
