"Which model should we use?" is one of the first questions in almost every engagement. The honest answer is usually "it depends on what you're building, and probably more than one." Here's how we actually think through the decision, rather than a leaderboard ranking that goes stale in a month.
The question underneath the question
Model quality leapfrogs every few months across every major provider, so a static "X is better than Y" comparison is close to useless by the time you read it. The more durable question is what dimension actually matters for your specific use case: raw reasoning quality, cost at your expected volume, latency, tool-use reliability, or context window size. Different models trade these off differently, and the right choice follows from your priority, not a general leaderboard.
The dimensions that actually matter
Reasoning quality on your specific task type
General benchmarks are a weak signal for how a model performs on your specific task. A model that's strong at creative writing isn't necessarily strong at structured data extraction or multi-step tool use. The only reliable way to know is testing against your actual use case with real examples, not trusting a general leaderboard score.
Cost at your actual volume
Price per token varies significantly across models and tiers, and at meaningful production volume it compounds into a real budget line, not a rounding error. A model that's marginally better but several times more expensive is often the wrong choice for a high-volume, lower-stakes task, even if it's the right choice for a low-volume, high-stakes one.
Latency
Some models and tiers are meaningfully faster than others. For a real-time voice agent or an interactive chat experience, latency can matter as much as raw quality. For an overnight batch processing job, it barely matters at all.
Tool-use and structured output reliability
For agentic systems that call tools and need reliably structured output, consistency matters more than peak creative capability. Some models are specifically stronger at reliably following a structured output schema, which matters enormously for production agent reliability.
Context window
If your use case needs to reason over long documents or extended conversation history, context window size becomes a hard constraint, not just a nice-to-have. This can rule out otherwise-attractive options for specific use cases.
Why "pick one model" is often the wrong frame
Most production systems we build use more than one model, matched to task. A common pattern: a fast, cheap model handles routing and simple classification steps, while a stronger, more expensive model handles the reasoning-heavy step that actually needs it. This single design decision is often the biggest lever for cutting cost without sacrificing quality where it matters. It stays invisible if you only think in terms of "which model did we pick?"
How to actually decide, in practice
- Define what "good" means for your task first: accuracy on a specific metric, cost per request, latency under a threshold, or whatever matters most.
- Test candidate models against real examples from your actual use case, not generic prompts, ideally with a labeled evaluation set you can score consistently.
- Model the cost at your real expected volume, not a rough guess. A small per-token difference compounds fast.
- Consider a multi-model architecture if different steps in your pipeline have genuinely different requirements, rather than defaulting to one model for everything.
- Re-evaluate periodically. Model capability and pricing both shift regularly, and a choice that was right six months ago may not be right today.
How we approach this
We build model selection into discovery, not as an afterthought. We map your actual task requirements, test candidates against real examples, and model cost at your real volume before committing to an architecture. When the evaluation supports it, we're comfortable recommending a multi-model design.