Benchmark
Translation quality, language by language
Held-out WASH test sets, 17 languages. Our engine — Gemma grounded in the WASH translation memory — against commercial MT and frontier LLMs. Filter languages, sort any column.
These are automated metrics. Scores come from a neural quality model (XCOMET-XL) and blind LLM judges — not human reviewers. They estimate quality; they don't replace professional evaluation.
Human validation, in progress. We're actively working with native-speaking domain/sector professionals who score the engine's output against human validators in our evaluation platform. Those expert ratings — not just automated metrics — will anchor future versions of this benchmark.
Reference-based — compares each translation to a human reference. The primary signal. Higher is better.
| Engine | MY | HA | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Our engine | 85.1 | 91.8 | 93.8 | 82.5 | 93.1 | 82.2 | 90.6 | 88.3 | 93.1 | 73.4 | 77.5 | 73.4 | 81.0 | ||
| Gemini 3 Flash | 84.6 | 90.8 | 93.4 | 82.0 | 92.2 | 82.7 | 90.5 | 88.0 | 91.3 | 73.0 | 77.1 | 74.2 | 79.7 | ||
| DeepL | 83.8 | 92.4 | 93.5 | 82.5 | 92.8 | 81.4 | 90.4 | 89.3 | 92.5 | 69.2 | 75.3 | 69.8 | 76.8 | ||
| Gemma 4 26B | 83.6 | 90.6 | 93.2 | 81.5 | 91.9 | 80.7 | 88.8 | 87.4 | 90.7 | 71.3 | 76.4 | 72.5 | 78.8 | ||
| Claude Sonnet 4.6 | 83.6 | 90.8 | 92.7 | 80.2 | 92.4 | 81.6 | 89.4 | 87.3 | 91.2 | 71.9 | 76.4 | 71.8 | 77.5 | ||
| Google Translate | 83.5 | 91.2 | 93.3 | 82.4 | 92.5 | 81.0 | 88.5 | 88.8 | 91.5 | 74.1 | 77.7 | 66.3 | 74.3 | ||
| Lara | 82.9 | 91.6 | 93.4 | 80.5 | 92.0 | 81.4 | 89.7 | 86.7 | 90.2 | 68.3 | 77.3 | 69.6 | 74.6 | ||
| ModernMT | 78.3 | 89.4 | 92.1 | 79.0 | 90.4 | 78.4 | 85.8 | 81.9 | 86.1 | 59.2 | 70.9 | 60.9 | 66.0 | ||
| Tiny Aya + TMX | — | — | — | — | — | 78.6 | — | — | 89.8 | 66.8 | 71.8 | 67.3 | 75.7 | ||
| Tiny Aya | — | — | — | — | — | 75.4 | — | — | 88.6 | 64.3 | 70.0 | 64.7 | 71.7 |
Tiny Aya (a second base model) was scored on a subset of languages — its rows show only where it ran and carry no corpus mean, so they stay out of the ranking. See the head-to-head below.
Burmese (my) and Hausa (ha) are shown but not yet scored — we're actively benchmarking them and will add their results soon. They don't count toward the means.
Second base model — Tiny Aya
Could a different base LLM beat Gemma? We ran Cohere Tiny Aya through the same translation-memory retrieval as our engine — a clean base-swap — and scored it head-to-head. Gemma + TMX still wins every language.
| Engine | id | sw | ne | hi | ur | bn | am |
|---|---|---|---|---|---|---|---|
| Our engine | 93.1 | 82.2 | 81.0 | 77.5 | 73.4 | 73.4 | 78.1 |
| Gemma 4 26B | 90.7 | 80.7 | 78.8 | 76.4 | 72.5 | 71.3 | 71.4 |
| Tiny Aya + TMX | 89.8 | 78.6 | 75.7 | 71.8 | 67.3 | 66.8 | 65.1 |
| Tiny Aya | 88.6 | 75.4 | 71.7 | 70.0 | 64.7 | 64.3 | 60.3 |
Gemma + TMX beats Tiny Aya + TMX in every language. The memory's lift is base-independent, but Gemma is the stronger pairing — so it stays our engine.
Tiny Aya + TMX still beats plain Tiny Aya everywhere (+1.1 to +4.8 XCOMET): retrieval helps whatever base it grounds. If a future base model overtakes Gemma, this is where it will show up first.
Cohere Tiny Aya (3.35B) · XCOMET-XL (reference-based), ×100, higher is better · added 2026-07-18.
COMET-blind languages — Akan and Haitian Creole
The COMET metric can't score these (its backbone has no Twi or Creole coverage): every engine clusters near the metric floor and can't be told apart. They're judged separately by large language models ranking blind — lower mean rank is better.
Akan / Twi
Akan is off-distribution for the COMET metric too, so it's scored by the same two blind LLM judges over 255 segments. Both judges self-reported competence and reasoned about real Twi specifics, and they agree strongly (Spearman ρ = 0.83).
| Engine | Claude Opus 4.8 | Gemini 3.1 Pro | ||
|---|---|---|---|---|
| Mean rank | Win % | Mean rank | Win % | |
| Gemini 3 Flash | 1.79 | 53% | 1.44 | 68% |
| Google Translate | 2.16 | 33% | 2.17 | 26% |
| Lara | 3.36 | 5% | 3.35 | 4% |
| ModernMT | 4.60 | 4% | 4.75 | 1% |
| Claude Sonnet 4.6 | 4.80 | 2% | 4.88 | 2% |
| Our engine | 4.94 | 2% | 4.94 | 0% |
| Gemma 4 26B | 6.24 | 0% | 6.37 | 0% |
Gemini wins Akan decisively and our engine sits mid-pack — but it clearly beats its own base model (about 1.3 ranks better), so the translation memory still lifts a weak base here. As with Haitian Creole, closing the gap is the base model's job; we show it rather than hide it.
Haitian Creole
Haitian Creole is off-distribution for the COMET metric backbone, so it's scored separately by two large language models judging blind. Each ranks every engine's output per segment; lower mean rank is better, and win % is the share of segments where it placed first.
| Engine | Claude Opus 4.8 | Gemini 3.1 Pro | ||
|---|---|---|---|---|
| Mean rank | Win % | Mean rank | Win % | |
| Gemini 3 Flash | 3.39 | 27.5% | 3.16 | 34% |
| Google Translate | 3.82 | 16.3% | 3.58 | 17.6% |
| Lara | 3.90 | 15% | 3.83 | 16.3% |
| DeepL | 4.18 | 12.4% | 4.37 | 6.5% |
| Gemma 4 26B | 5.47 | 5.9% | 5.90 | 4.6% |
| Claude Sonnet 4.6 | 5.48 | 7.2% | 5.60 | 3.9% |
| Our engine | 5.61 | 7.8% | 5.47 | 7.8% |
| ModernMT | 5.99 | 3.9% | 5.92 | 5.9% |
| Kreyòl-MT(specialist) | 7.16 | 3.9% | 7.18 | 3.3% |
Both judges agree Gemini leads the field, and our engine sits mid-pack on Haitian Creole. Retrieval over the thin Haitian Creole memory gives little lift here — base Gemma and Gemma + TMX land within a few tenths of a rank of each other — so this gap is the base model's, not something the memory can yet close. We show it rather than hide it.
How it was measured
- XCOMET-XL. A 3.5B-parameter neural metric (WMT 2023 SOTA) that scores a translation against a human reference, modelling adequacy and error spans. Shown ×100.
- The engine. “Gemma + TMX” is base Gemma 4 26B grounded in the WASH translation memory via retrieval — the product. “Gemma 4 26B” is the same base model without retrieval, included to isolate what the memory adds.
- Dataset
- baobabtech/tmx-benchmark-results
- Sample
- 153–300 segments per language
- Generated
- 2026-06-27