Ask most teams how they chose their model, and the honest answer is: whichever one they started with.
That is usually a large model. It got the demo working, so it stays in production long after anyone has asked whether it is still the right tool for the job.
The question usually gets framed as: should we use a small model or a large one?
That is the wrong question. The right one is:
What is the least complex model architecture that can meet the required level of performance, reliability, speed, control and economic sustainability in production?
For most production systems, the honest answer will not be one model at all. It will be a hybrid: deterministic code handling what is predictable, a small model handling narrow work at volume, a large model catching what is ambiguous, and a person accountable for what matters most.
Getting there takes working through the decision deliberately, instead of by inertia.
Earn the right to specialise
A small language model is not a cheaper large language model. It is a different bet: narrower capability in exchange for lower cost, lower latency and more control. That bet only pays off under specific conditions.
Use a small model when the task is narrow enough to specialise, frequent enough to justify optimising, and measurable enough to govern. Concretely: the scope is bounded, the operation repeats, correctness can be judged against agreed criteria, the volume makes efficiency worth pursuing, and the organisation can actually deploy, monitor and maintain the model once it is live.
A large model tends to remain the better fit where the work is open-ended, ambiguous, infrequent, fast-changing, or dependent on reasoning across domains — an advisory task where the conversation moves unpredictably between commercial, technical and operational questions, for instance, or a planning task where the right reasoning depends on circumstances nobody anticipated at design time.
Moving that kind of work to a small model is not automatically progress, even when the small model performs adequately in testing. Splitting one workload across several specialised models can simply move the complexity somewhere else: more endpoints, more deployment pipelines, more versions, more evaluations, more things somebody has to own. A model that is cheaper per request can still make the overall system more expensive to run. That is why the comparison has to be made at the level of the operating model, not the model card.
The objective is not the smallest model you can get away with. It is the smallest model that clears the production bar. We made a version of this same argument about autonomy: use the least autonomous architecture capable of delivering the outcome. Model size is the same discipline applied one layer down.
Six questions before you touch a model card
Before comparing benchmarks, work through six questions in order. Skipping one is how a model decision ends up looking sound on paper and failing in production.
1. Business value.
What process, service or decision improves, and how often does it run? A narrow task performed rarely will not justify the cost of specialising a model, however clean it looks in isolation.
2. Task shape.
Is the scope fixed? Are inputs consistent, outputs structured? A useful test: could a capable person describe the task, its inputs, its outputs and its main failure modes on one page? If not, it is not yet bounded enough for a small model.
3. Required performance.
Not “how good is the small model in general” but “does it clear the threshold on this task, on this organisation’s own data.”
4. Deployment and control.
Does the task have a genuine requirement for local execution, low latency or reduced cloud dependency — or is that simply a technically appealing option nobody has stress-tested?
5. Full economics.
Is the small model cheaper once evaluation, infrastructure, monitoring, security, retraining and the large-model fallback it will still occasionally need are all in the total?
6. Organisational readiness.
Who owns the model after deployment, and what happens when its performance drifts? No clear answer means no small model yet.
Work through these honestly and a pattern usually appears fast. Narrow, high-volume, well-instrumented: strong candidate. Broad, low-volume, poorly measured: not yet, regardless of how well the demo went.
Test the threshold, not the average
The comparison that matters is not how the small model performs against the large one on general benchmarks. It is whether the small model meets the bar this specific task requires.
Build a task-specific evaluation set from real production examples — normal cases, hard cases, known failure modes. At minimum, evaluate accuracy, consistency across inputs, instruction adherence, format compliance, latency, and whether low-confidence or invalid outputs can be caught at all. Compare at least three options: the current or proposed large model, an off-the-shelf small model, and a tuned version where one is justified.
A model that is cheaper per call but regularly produces outputs someone has to fix is not cheaper. It has just moved the cost from the inference bill to somebody’s afternoon.
Cost the operating model, not the inference bill
Lower cost per token does not automatically mean lower total cost.
A large hosted model brings API usage, token volume, rate limits, integration, monitoring and vendor dependency. A small model brings model selection, infrastructure or device capacity, deployment engineering, evaluation, monitoring, security, retraining — and the large-model fallback calls it will still need for the cases it genuinely cannot handle.
There is a real counterargument worth taking seriously here: a centrally hosted large model can achieve better infrastructure utilisation than a scatter of specialised endpoints. A small model can be cheaper per inference while still costing more overall, if demand never reaches the volume that justifies running it.
Run the comparison across more than one scenario — current volume, expected volume, ten times current volume, lower-than-expected adoption. The useful question is not “is the small model cheaper today.” It is “at what volume does it become cheaper, and are we realistically going to reach it.”
Score it before you commit
The six questions above tell you whether a small model is worth investigating. This turns that judgement into something you can actually compare across candidate tasks, rather than re-litigating the same six questions for every workload. Score each dimension from 1 (weak case) to 5 (strong case):
|
Dimension |
Weak case (1) |
Strong case (5) |
|---|---|---|
|
Task boundedness |
Broad, unpredictable |
Narrow, stable |
|
Volume and repetition |
Low or occasional |
High and recurring |
|
Performance feasibility |
Small model well below threshold |
Small model meets threshold |
|
Latency or edge need |
No material requirement |
Operationally essential |
|
Economic case |
No demonstrated saving |
Clear break-even, material benefit |
|
Operating readiness |
Limited internal capability |
Strong ownership in place |
Above roughly 24 out of 30: a strong candidate for a dedicated small-model evaluation. In the middle: a hybrid, with the small model handling standard cases and a fallback for the rest. Below roughly 15: keep the large model, or redesign the workflow before touching model architecture at all. The score should prompt discussion, not replace evidence — nothing goes into production on a score alone.
Build a routing architecture, not a single model
The workload rarely needs one model end to end. It needs a hierarchy.
Deterministic code for anything with stable, explicit rules: validation, calculations, permissions, workflow state. A small model for narrow language tasks: classification, extraction, re-ranking, structured summarisation, tool selection. A large model for what genuinely requires broad understanding, ambiguity resolution or multi-step reasoning. Human review where the consequences, uncertainty or novelty justify it.
Code first where you can. A small model by default where it is sufficient. A large model when it is necessary. A person when it is warranted.
The migration path, if you already have an LLM in production
Most organisations do not choose between small and large models on day one. They start with a large model to learn how the workflow actually behaves, then use the resulting production data to find what is worth specialising.
The pattern holds across most workloads we see:
- Instrument what you already have. Log task type, input, output, latency, cost, tool calls, human corrections and failure reason.
- Cluster the workload. Group requests into recurring categories rather than treating every call as unique — classification, extraction, summarisation, routing, exception handling.
- Prioritise candidates. Look for high volume, stable patterns, measurable output, meaningful large-model cost, and low reasoning complexity.
- Set the baseline. Measure the current model’s quality, latency, cost and failure rate on that task specifically, before touching anything.
- Test an off-the-shelf small model before building one. Fine-tuning is a later step, not a first one.
- Adapt only where the gap justifies it, and only with enough representative data to do it properly.
- Route with a fallback, sending anything low-confidence, out of domain, or flagged as high-risk back to the large model or a human.
- Monitor and expand deliberately. Pass rate, fallback rate, latency, drift, cost per successful outcome. Grow coverage only where the evidence supports it.
Different organisations, different starting points
The right default is not the same everywhere.
Enterprises with large recurring workloads, mature data capability and meaningful inference spend have the strongest case for specialisation — and the biggest risk of an uncontrolled sprawl of small models with no shared standards or ownership behind them.
Public-sector organisations often have a genuine case for local or sovereign deployment and strict auditability. Running a model locally does not remove the need for governance, access control and human oversight. It just relocates where those controls need to sit.
SMEs should be cautious about assuming self-hosting is automatically cheaper. Without the volume or the specialist maintenance capacity, a well-instrumented hosted large model is often the more sensible starting point, with selective specialisation once a stable, high-volume pattern has actually shown up in the data.
The small-model gate
Before approving a small model for production, the answer should be yes to each of these:
- Is there a measurable reason to move away from the large model, with a realistic break-even point behind it?
- Is the task narrow and stable, with identifiable exceptions?
- Has it been tested on representative data against an agreed threshold, with low-confidence outputs detectable?
- Has the workflow been decomposed, with a defined fallback, rather than handed wholesale to one model?
- Are logging, monitoring, ownership and an incident process in place?
- Are data access, deployment boundaries and human oversight proportionate to the consequences?
Several “no” answers is not a reason to abandon specialisation. It is a reason to keep the large model, redesign the workflow, or run a controlled pilot first.
The bottom line
In most mature production systems, the answer is neither “small model” nor “large model.” It is a governed hybrid that assigns each task to the least complex model capable of performing it reliably, with deterministic code and human review filling in where they are simply the better tool.
That is the same discipline running through how we think about AI economics and production architecture more broadly: start with the business outcome, design the complete system around it, build in the governance the consequences warrant, and let production evidence decide what runs where — not a benchmark leaderboard.
At ALGO, model selection is one part of a wider discipline that includes the software engineering, data, cloud, DevOps, cybersecurity, monitoring and operational support needed to keep a production AI system reliable after launch, not just impressive at launch.
Because the goal was never to pick the biggest model, or the smallest one.
It was to build a system that does the job.
Want to discuss your model architecture?
If you are running a single large model across a workload that has outgrown it, or trying to work out whether a smaller, specialised model would hold up in your production environment, get in touch.
Email: info@algocodingexperts.com
Phone: +34-91-633-1884
Contact form.
About ALGO
ALGO is an engineering partner specialising in advanced technologies. We help startups, SMEs, enterprises and public-sector organisations design, develop and integrate technically demanding systems built for the real world. Our work combines specialist technology expertise with the software, data, cloud, DevOps, cybersecurity, UX, QA, integration, monitoring and operational capabilities required to turn advanced technology into complete, reliable and sustainable systems.