We tested every open Amharic speech model, then made the best one better

We tested every open Amharic speech model we could find on data none of them had ever seen. Then we built a 538 MB file that makes the winner measurably better, and put the whole thing online so you can talk to it.

If you want to build something that listens to Amharic, you have to pick a model. Ask which one is best and you get download counts, model card claims, and numbers measured on whatever data the author happened to have. None of that tells you what will happen to your users.

So we measured it. Five open speech recognition models, on 1,548 recordings, scored the same way.

Why you can believe the numbers

Two things make this test unusual, and both matter more than the ranking itself.

We know exactly what was said. Every recording is a person reading a sentence we showed them. The text existed before the audio did. Most speech benchmarks compare against transcripts someone typed while listening, and typists make mistakes that nobody counts. Here there is no typist to be wrong.

None of the models could have studied for it. Our test set was published in August 2026. Every model in the table was published between five and nine months earlier. You can check the dates yourself. This sounds like a small thing and it is not: if a model has already seen the answers, you are measuring its memory, not its hearing.

The results

Imagine a typical Amharic sentence of a hundred characters. Here is roughly how many characters each model gets wrong.

Nine wrong characters in a hundred sounds bad until you read the output. Most of those errors are a vowel marker on the wrong form of a letter, the kind of thing a reader corrects without noticing. The sentence usually survives. At twenty seven, it does not.

Three things we did not expect

The small model beat the big one. Two models from the same family, one with 606 million parameters and one with 963 million. The smaller one is noticeably better and cheaper to run. We had spent two weeks building on the larger one because we assumed size ordered quality. It does not.

Specialising beat generalising. The two best models are the same architecture at the same size, 606 million parameters each. The only difference is that one was trained on Amharic alone and the other on five Ethiopian languages, and the Amharic-only one wins. Sharing training across related languages is supposed to help low resource languages. For Amharic, which has more data than its neighbours, it costs a little instead.

Two fair metrics disagree about who wins. Count mistakes by character and the Ethio-ASR 600M model wins. Count them by whole word and Shook wins. Neither is cheating. Shook produces fluent, complete words that either match or do not, while the other model gets closer letter by letter but leaves more words slightly off. Amharic glues prefixes and suffixes onto stems, so one wrong letter can spoil a long word. Which metric you pick decides your winner, and most papers only tell you one.

The fastest one is also the most accurate

We timed every model on the same hardware. The differences are much larger than the accuracy differences, and they point the same way.

The most accurate model is also six times faster than the next most accurate one. There is no trade to make at the top of that table, which is unusual and worth knowing before you pick.

The stranger result is that the 38 million parameter model is slower than the 606 million one. Size is not what decides this. The fast models read the whole recording in one pass. The slow ones write out the answer one character at a time, so they pay for every letter they produce. If you are transcribing thousands of hours, that difference matters far more than parameter counts suggest.

So we made the best one better

A speech model like the winner here works one sound at a time. It listens to a twentieth of a second, decides which letter that was, moves on. It never steps back and asks whether the letters it just produced spell an actual word.

That is why its mistakes look strange. Not wrong words, but near-words: letter sequences that sound about right and mean nothing.

The fix is old and cheap. Train a second model on written Amharic alone, one that knows nothing about audio and only knows which words tend to follow which. Then let it vote on the speech model's guesses.

We built one from 12.8 million lines of Amharic text. The first version was 17 GB, which nobody can download, so we cut it down and measured what each cut cost.

The 538 MB version does what the 17 GB version does. Thirty-two times smaller for a difference too small to measure. Three words in every hundred, fixed by a file you could email.

Then we tried it on ourselves

Benchmarks are people reading prepared sentences. So we recorded ourselves talking normally, mixing in English the way everyone actually does in Addis, and watched what happened.

The Amharic came out clean. The language model tidied a name and removed a stray full stop.

Then it broke the English. The speech model had heard my name is and written it out phonetically in Ge'ez, which was right. The language model had never seen an English word in its life, decided that could not possibly be Amharic, and helpfully corrected it into something else.

It was confidently fixing a word that was already correct. Nothing in the benchmark could have caught this, because the test sentences come from news text and contain almost no English. It took one person talking normally for thirty seconds.

That is now written down as a known limitation, and it points somewhere useful: the next language model should be trained on Amharic as it is actually spoken, English and all.

You can try it right now

We put it online. Speak into it, or upload a clip, and you get the transcript twice: once from the speech model alone, once with the language model correcting it. Watching the difference is the whole point.

There is also a button to tell us whether it got your words right. That matters more than it sounds. Every open Amharic speech model is measured on people reading prepared sentences, because no one has labelled data for real conversation. Your verdict on your own words is exactly the thing that does not exist.

What went wrong on the way

Benchmarks are published as though they arrived fully formed. This one took four days, and most of that was spent being wrong.

The worst bug gave Shook a score three times worse than the truth. We had capped how much text the model was allowed to produce, so on long sentences it stopped mid-way and was scored for words it was never allowed to say. Nothing crashed. The number just looked plausible and was completely wrong.

What caught it was not the score. It was a boring check we almost skipped: comparing how long each model's output was against how long it should have been. On short sentences Shook produced 96 percent of the expected text. On long ones, 46 percent. A model that is merely wrong is wrong at every length. A model being cut off gets worse as sentences grow.

There were nine others. A shutdown timer we triggered by trying to switch it off. A health check that declared a working tool broken because the tool exits with an error code when you ask it for help. An image library four versions out of date, which reported itself as a completely unrelated import failure and cost us two machine launches. Every result silently failing to save because a stray environment variable pointed at the wrong account.

We wrote all ten up, with what each one cost and what fixed it, because three of them produced results that looked correct. That is the part worth reading if you do this kind of work.

What this does not tell you

Everyone in this test is reading a prepared sentence into a phone. One speaker, no background noise, no interruptions, proper grammar. That is not conversation.

We checked what happens on real podcast audio, and these same models disagree with each other about three times as much there. They are all substantially worse when people actually talk. So treat these numbers as the best case, not the expected case, and do not let anyone quote them as what Amharic speech recognition can do.

We are building a spontaneous speech test set next, which is the number the field actually needs and nobody has.

Go and use it

The demo takes your voice and shows the transcript with and without the language model.

The language model is 538 MB and free. If you are building anything that listens to Amharic, it is three words in every hundred you were about to get wrong.

Take it and check us

Everything is public: the scores, the method, the full failure log, the code, and every model's raw output for all 1,548 recordings. If you think a decoding choice we made was unfair to your model, you do not have to argue about it. Rerun it.

huggingface.co/datasets/snapwre/amharic-asr-benchmark