On the text leaderboard of Arena, a public scoreboard that ranks AI models by user votes, the top three scores sat about five points apart at the time of writing [1]. The site's own margin of error, its estimate of how far off each score could be, was about five points too. Read strictly, the top of the table was a tie, printed as a ranking all the same. That is the starting problem for anyone working out how to choose an LLM.So if the ranking can't pick for you, how do you actually choose the right LLM for your use case?
The method has two steps, and most teams only ever run the first: build a shortlist from the public leaderboards, then decide with a test on your own work. Hiring is the closest everyday version. The leaderboard is the résumé pile with grades on it, and nobody sane hires from the pile alone. Your own test is the work sample you hand the finalists.
What LLM leaderboards do well
The shortlist step is résumé-pile work, and the boards do most of it for you. They tell you which models are in the right league at all. They also let you cross names off: when one model beats another on both measured quality and measured cost, the beaten one is done. The big boards even ship filters for price and other practical constraints next to the ranking for exactly this purpose [1]. And they flag newcomers, since an unfamiliar name near the top usually deserves a closer look.
Where the ranking stops answering
Two things stop the shortlist from doubling as the decision. The first is that the overall ranking is an average, and averages hide the differences a task-level choice needs. When I checked the board, the model on top overall sat third in coding [1]. Another model sat ninth overall, yet it placed third in creative writing and twenty-first in coding. Averaged across everything, that second model looks middling, yet for writing it was one of the best available.
The second is that benchmarks wear out. The people who run them say so themselves. A benchmark is a fixed set of test questions used to grade models, and a fixed set ages badly, because the questions leak into the material models train on. In February 2026, OpenAI stopped reporting results on a coding benchmark it had used since 2024 [2], writing that gains there "no longer reflect meaningful improvements in models' real-world software development abilities". Its audit found flaws in more than half of the hard problems it checked, and the models it tested could repeat the official answers word for word, a sign they had seen the test during training.
A leaderboard can tell you which models are worth testing. It can't tell you which one wins your task, because nobody on it ever voted on your work. Step one should end with a shortlist of three or four models: one frontier model (a big flagship system), one or two mid-priced options, and one small model you half expect to fail. The small one is there to break first, because where it breaks tells you how little model your task really needs.
How to evaluate LLMs on your own tasks
Your own test sounds like a quarter-long project. The useful version starts in an afternoon. Say your team turns customer emails into short summaries for support agents. Pull twenty real emails, including the five your current model fumbled last month. Before running anything, write down what a passing summary must contain: the customer's request, the order number if one appears, and nothing invented. Run every shortlisted model over the same twenty emails, mark each output pass or fail, and note the tokens each spends (tokens are the small chunks of text vendors bill by). That is an afternoon of work, and at the end of it you know how each model handles your actual inbox.
That scale is also what one model maker tells its own customers. Anthropic, the company behind the Claude models, puts the start at "20-50 simple tasks drawn from real failures" [3], the twenty-email exercise with room to grow. Its rule for a good test item is that two people who know the work should reach the same pass or fail verdict on their own; if they could disagree, tighten the pass rules or drop the task. And keep the token counts, because the scoring step needs them.
The score is cost per completed task
Suppose three models clear your bar. Which should win? One model answers in three sentences; another reasons at length first, and every extra word is billed. So price each run: tokens spent, times the price per token, for one finished job. That unit is called cost per task, and Artificial Analysis, a company that measures AI models, likewise prices whole completed tasks [4]. The case for the unit, with real models and real prices, is the first article in this series. Put pass verdicts and per-task prices side by side and the decision mostly makes itself: the winner is the cheapest model per completed task that clears your bar.
Few teams ever build that table. LangChain, an AI developer-tools company, surveyed 1,340 respondents and found that just over half run structured tests of their own [5]. Far more of them just watch how the model they already chose behaves in production, meaning live on real work, and watching never shows what a different model would have done with the same tasks. Keeping that comparison running across models in production takes software that sits between your applications and the model vendors, a control layer such as Hyntelo AI Gateway.
Expect more than one winner
Expect your table to end without a single champion. When Artificial Analysis measured what models know across six subject areas, models from three different labs each led somewhere [6], and its own conclusion was to choose by use case rather than by overall rank. The leaderboard's category splits said the same. A twenty-task test is small and will miss things. It still beats choosing from a public ranking alone, because its misses happen on work you can see. Nobody expects one hire to fit every open role, or one model to win every task.
If different tasks have different winners, and your own numbers now prove it, why does every task in production still go to one model?
References
[1] Arena.ai, "Text leaderboard", captured August 19, 2026, https://arena.ai/leaderboard/text
[2] OpenAI, "Why SWE-bench Verified no longer measures frontier coding capabilities", February 2026, https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
[3] Anthropic Engineering, "Demystifying evals for AI agents", January 2026, https://anthropic.com/engineering/demystifying-evals-for-ai-agents
[4] Artificial Analysis, "Methodology", accessed August 2026, https://artificialanalysis.ai/methodology
[5] LangChain, "State of Agent Engineering", December 2025, https://www.langchain.com/state-of-agent-engineering
[6] Artificial Analysis, "AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models", arXiv:2511.13029, November 2025, https://arxiv.org/abs/2511.13029
