An open-access, scientifically rigorous evaluation surface mapping the performance and agentic capabilities of frontier models across non-English tasks, socio-cultural contexts, and multimodal domains.
Standard evaluation frameworks mask the performance degradation of frontier systems when deployed outside English-native workflows. LILT Applied AI stress-tests LLMs under realistic, localized constraints to map the true boundaries of global AI capability.
Task execution accuracy across our multilingual suite.
| Language | % tasks revised | Word-level edit rate | Char-level edit rate |
|---|---|---|---|
| Arabic | 84.8 | 25.4 | 19.4 |
| German | 81.2 | 30.0 | 22.2 |
| Hindi | 100.0 | 55.4 | 46.3 |
| Korean | 92.1 | 36.9 | 27.9 |
| Portuguese | 87.9 | 25.0 | 19.5 |
pass rate (equivalently pass@1 or pass^1) is the chance a model will successfully pass a single task. We estimate it by having each model complete each task multiple times.
pass@k is the chance that a model will pass a task at least once out of k attempts. Intuitively, this is best used for difficult tasks that could reasonably be solved by running the model multiple times and selecting the best answer. This is estimated as above, but with the use of the unbiased estimator (see Appendix A of Chen et al., 2021).
pass^k is the chance that a model will pass a task reliably in all k attempts. This metric is best used for tasks that can't be rerun (like interactions with a customer, as in τ³-bench) or where failure is expensive or dangerous. It is estimated using the unbiased estimator introduced in the first τ-bench paper (Yao et al., 2024).
Live results from the LILTBench task collective — community teams authoring coding tasks designed to break frontier models in their own language, scored against vertex_ai/claude-opus-4-6 over 15 iterations. Click a row to expand language coverage & scores.
Sourced from liltbench-tasks-public · updated 2026-06-24 13:32 UTC · click any header to sort