When two speech models agree, are they right? We measured it, and our first answer was wrong

We wanted to know whether two speech recognition models agreeing with each other means the transcript is correct. The experiment cost about a dollar. Getting a trustworthy answer took three more tries.

There is far more Amharic audio in the world than there is transcribed Amharic audio. Podcasts, radio, sermons, lectures: tens of thousands of hours, none of it labelled. If you could label even a fraction of it automatically, you could train far better speech recognition than the transcribed data alone allows.

The standard trick is to have two different models transcribe the same audio and keep only the parts where they agree. It is roughly how Whisper was built. But it rests on an assumption nobody usually tests: that agreement means correctness.

Normally you cannot test it. If you had the right answers you would not need the trick. We can test it, because our corpus is people reading prompts aloud, so we already know what every recording says.

Two models worth using

We picked two open models trained specifically on Ethiopian languages, by two different people, on different data, with different architectures.

Shook Medium is a Whisper fine-tune on roughly 2,000 hours of Amharic, released openly by Addis AI. Ethio-ASR is a w2v-BERT CTC model covering five Ethiopian languages, released with a paper by Badr al-Absi. Different architectures trained by different teams is the point: models that fail the same way agree with each other for the wrong reasons.

We deliberately did not use Whisper or Meta's MMS, which the obvious version of this pipeline would reach for. Both are poor at Amharic. MMS scores 52.0 word error rate on our test set, and the Ethio-ASR paper measured Whisper and SeamlessM4T above 100% on Ethiopian languages, which means they insert more than they transcribe.

The first answer was wrong

We ran both models over 500 recordings where we knew the correct transcript, and measured how well model-to-model agreement predicted actual correctness. The result looked excellent. Rank correlation of 0.935. Agreement predicted correctness almost perfectly.

It was measuring a bug in our own code.

We had capped how much text Shook was allowed to produce, and the cap was too low for long Amharic sentences. Its transcripts were being cut off, and the longer the recording the worse it got. On recordings over twenty seconds, Shook was returning less than half the text it should have.

So "disagreement" was not measuring whether the two models were both wrong. It was measuring how badly one of them had been truncated.

The number that caught it

We measured the same thing twice, from each model's side. That is what exposed it.

In the first run, agreement predicted Shook's errors at 0.935 and Ethio-ASR's at 0.087. Those two numbers should be similar. A gap that large says agreement is tracking one model's failures rather than mutual corroboration. The third row is the confirmation: agreement correlated with recording length at 0.824, because length was what triggered the truncation.

After the fix, the two model numbers sit close together at 0.685 and 0.736, and the length effect collapses to 0.108. The headline number went down and the result became trustworthy.

What the models actually score

With the bug fixed, here is the first published measurement of either model on our test set, which is ordinary people recording on ordinary phones rather than studio audio.

We use character error rate rather than word error rate because Amharic is agglutinative. One wrong suffix on a long word counts as an entirely wrong word, which throws away most of the signal.

Shook is more accurate on average. Ethio-ASR is more reliable: it never once produced a badly wrong transcript, while Shook did six times. They are also complementary, with Shook better on 61% of recordings and Ethio-ASR on 29%, so picking the better of the two per recording would beat either alone.

The answer to the question

Yes, agreement predicts correctness, and here is the price list.

Keeping only recordings where the two models agree closely cuts the label error roughly in half, from 0.091 to 0.052, while still keeping 58% of the data. That is a good trade, and it is now the setting we are using on unlabelled audio.

One thing that turned out not to matter

Amharic writes some syllables two ways. ሟ can be a single character or spelled out as ሙ followed by ዋ, and both are correct. We expected the two models to disagree constantly over this and inflate the error, so we built a normaliser to fold the variants together and measured how much it changed the result.

It changed 11 recordings out of 500, and moved the average by 0.0001. So we are not adding it to the published evaluation script. Leaving the metric alone is worth more than a rounding error, and we would rather report a measurement than an assumption.

Why we are writing this up

The first run produced a number that agreed with what we expected and was wrong. It would have passed review. It would have looked fine in a paper.

What caught it was measuring the same quantity two ways and noticing they disagreed. That is a cheap habit and it is the only reason we are not currently building a pipeline on a broken premise. If you take one thing from this, take that rather than the thresholds.

The whole experiment, including three failed runs, cost about $1.50 of GPU time. The expensive part of research like this is usually not the compute. It is having a test set that can tell you when you are wrong.

Where this goes

Calibrating the filter was the point of the exercise. The reason to calibrate it is to use it on audio nobody has transcribed, and the reason to do that is to train a speech recognition model on Ethiopian data and publish it openly. That is the direction. We will write up what happens, including if it does not work.

Everything above can be checked. The test set, the evaluation script and the homophone folding all ship inside the dataset, so anyone can reproduce these numbers or show us where we are still wrong.