Do You Really Need an LLM for Enterprise AI? When Smaller Models Are Enough

Enterprise AI conversations often start with the same assumption:

If we want the best AI, we need the biggest model.

That assumption made sense when large language models were the easiest way to get broad reasoning, general knowledge and strong language capabilities into an application.

But enterprise AI is becoming more specialised.

A customer support system may need to classify an intent. A banking workflow may need to identify a transaction type. A healthcare application may need to extract structured information. An internal system may need to route a request to the correct workflow.

These are important tasks, but they do not necessarily require a large general-purpose LLM.

The more useful enterprise question is:

What level of intelligence does this particular task actually need?

That shift changes how companies think about model selection, latency, inference cost, token consumption, deployment and AI architecture.

Microsoft Research recently argued that many enterprise workloads are deterministic, structured and knowledge-dependent, and that using large language models as monolithic engines for every such task can be inefficient. Its position is that enterprises can benefit from architectures that externalize knowledge and computation into dedicated components rather than asking the language model to do everything.

AWS has reached a similar architectural conclusion in its current guidance for agentic systems: model capability should be matched to task complexity, with smaller models used where they meet the required quality bar and larger models reserved for tasks that genuinely need them.

For many enterprise applications, the answer is not LLM or SLM.

It is LLM where it adds value, SLM where specialization is enough, knowledge where knowledge is needed, and deterministic systems where rules are enough.

What does an enterprise actually need an LLM for?

Before choosing a model, it helps to look at what the application is actually asking the AI to do.

An enterprise workflow can contain several different types of tasks.

Language understanding

The system may need to identify:

• User intent
• Entities
• Language
• Sentiment
• Workflow state
• Customer request type

These tasks are often narrower than open-ended reasoning.

Structured extraction

The system may need to turn unstructured language into structured data.

For example:

“The customer wants to change the EMI date to the 15th.”

The system may need to extract:

intent = change payment date

date = 15th

That does not necessarily require a model capable of answering arbitrary questions about the world.

Classification and routing

An enterprise application may receive millions of requests that need to be routed to:

• billing
• technical support
• sales
• returns
• KYC
• claims
• human escalation

Again, the task has a defined output space.

Workflow decisions

The AI may need to determine what should happen next inside a known process.

That could mean:

verify identity → check eligibility → retrieve policy → perform action

The workflow itself can remain deterministic, while a smaller model handles the language understanding needed to move through it.

Complex reasoning

This is where larger models become much more valuable.

A system may need to:

• compare competing evidence
• synthesise information across multiple sources
• solve an unfamiliar problem
• reason through ambiguous requirements
• generate a detailed analysis
• handle open-ended questions

These are the workloads where a more capable general-purpose model can justify its additional cost and latency.

The mistake is using one model for everything

Imagine an enterprise application where every request goes to the same large model.

A simple request:

“Where can I download my tax statement?”

A moderately complex request:

“Which policy applies to this customer’s account?”

A complex request:

“Analyse these five contracts and explain the financial and operational risks.”

All three may be sent to the same model.

But they are not the same problem.

The first may need intent detection and retrieval.

The second may need knowledge grounding and business rules.

The third may genuinely require complex reasoning.

A single-model architecture treats all three as equally difficult.

A more efficient architecture asks:

What does this request actually need?

This is the basis of model routing.

Microsoft’s current model-router documentation explicitly describes routing prompts to different models based on complexity, reasoning requirements and task type, with cheaper and faster models used for simpler requests and higher-capability models reserved for more demanding prompts.

What can smaller AI models handle?

A Small Language Model can be particularly useful when the task is narrow, repetitive, measurable and domain-specific.

Common examples include:

1. Intent classification
   Determine what the user is trying to do.
2. Entity extraction
   Extract names, IDs, products, dates, amounts or other structured information.
3. Routing
   Send requests to the right team, workflow or system.
4. Domain terminology
   Understand specialised enterprise vocabulary.
5. Structured generation
   Produce predictable JSON or other structured outputs.
6. Workflow state detection
   Determine where a customer or process is in a predefined journey.
7. Policy matching
   Identify which policy or process applies.
8. Voice-agent state handling
   Track whether a caller is verifying identity, making a payment, requesting support or moving to escalation.
9. High-volume classification
   Handle large numbers of similar requests with lower inference overhead.

These are not trivial tasks.

They are just well-defined tasks.

That distinction matters.

Why an SLM can be a better fit for a specialised task

A general-purpose LLM has to be capable across a huge range of possible inputs.

A custom SLM can be trained around a much narrower problem.

Shunya’s Custom SLM approach focuses on purpose-trained models built around enterprise terminology, workflows, intents and decision-making rather than expecting one general-purpose model to handle every task.

Its current published benchmarks report 10× smaller models, 3× faster inference and 94% accuracy retention for its evaluated workloads. Shunya also reports 240+ concurrent users on a single L4 and approximately 20× lower inference cost for an equivalent workload in its published benchmark comparison. These are Shunya-reported benchmark results and should be evaluated against the specific workload and methodology rather than treated as universal performance claims.

The important idea is not that smaller is automatically better.

It is that specialisation can remove unnecessary model complexity.

Smaller models do not need to memorise the entire business

One reason enterprises often reach for large models is the belief that the model needs to somehow “know everything.”

It does not.

The model can work alongside a dedicated knowledge layer.

For example:

Customer → owns → Account

Account → uses → Product

Product → governed by → Policy

Policy → applies to → Region

A knowledge graph can represent those relationships explicitly.

RAG can retrieve supporting documents.

APIs can provide current operational information.

Business rules can handle deterministic decisions.

The model can then focus on understanding the request and performing the language task.

Shunya’s enterprise AI platform describes this architecture around a knowledge graph containing enterprise concepts such as products, policies, SOPs and workflows, combined with dedicated models and business logic.

This is also why the architecture matters more than parameter count alone.

A model does not have to carry the entire company’s knowledge inside its weights if the system can give it reliable, relevant context when it needs it.

An enterprise AI stack does not have to be model-centric

A more modular architecture might look like:

User

↓

Intent / task detection

↓

Knowledge layer

• Knowledge graph
• RAG
• Enterprise search
• Structured data

↓

Model selection

• Small specialised model
• Larger reasoning model when required

↓

Business rules

↓

Enterprise systems

• CRM
• ERP
• Databases
• APIs
• Workflow systems

↓

Response or action

This is broadly consistent with the architecture approach Shunya describes across its enterprise platform and Custom SLMs.

The benefit is separation.

Knowledge is handled by the knowledge layer.

Rules are handled by deterministic systems.

Specialised language tasks are handled by smaller models.

Complex reasoning is handled by larger models.

The result is a system that does not ask one model to do everything.

What tasks actually need a large LLM?

A larger LLM still has an important role.

There are many situations where a small model is not the right choice.

Broad reasoning

If the model needs to combine many different concepts and reason through an unfamiliar problem, a larger model may be appropriate.

Open-ended research

Research tasks often involve unpredictable questions and diverse information sources.

Complex synthesis

A system may need to compare several documents and produce a detailed analysis with competing considerations.

Unstructured decision support

Some business problems cannot be reduced to a fixed set of categories or workflows.

Advanced generation

Long-form writing, nuanced drafting and creative generation can benefit from broader model capabilities.

Complex agentic workflows

Some agents need to plan across several steps, decide which tools to use, interpret their results and recover from unexpected situations.

These are precisely the situations where additional model capability can justify additional cost and latency.

The goal is therefore not to eliminate large models.

The goal is to stop using them where they are unnecessary.

The best enterprise architecture can use both SLMs and LLMs

Consider a customer-support system.

A customer says:

“My payment went through yesterday but my account still says pending. Can you check?”

The system might work like this:

Custom SLM → identify payment-status intent

↓

Knowledge graph → identify relevant account and product context

↓

API → retrieve current payment status

↓

Business rules → determine the valid next step

↓

LLM → only if a complex explanation or reasoning step is required

↓

Response

The large model may not be involved at all for many requests.

For a more complicated request, the system can escalate.

That creates a model hierarchy rather than a single-model dependency.

Model routing makes the architecture more practical

Not every request needs the same level of intelligence.

A routing layer can classify requests by:

complexity

risk

domain

latency requirement

data sensitivity

task type

The system can then select an appropriate model.

For example:

Request type	Recommended path
Intent classification	Custom SLM
Entity extraction	Custom SLM
FAQ lookup	SLM + RAG
Policy lookup	SLM + knowledge graph + RAG
Structured workflow decision	SLM + business rules
Complex multi-document analysis	Larger LLM
Open-ended research	Larger LLM
Creative generation	Larger LLM
On-device voice command	Tiny local model

This approach is already becoming more common in enterprise model architecture. AWS recommends mapping different task-complexity classes to the smallest model that meets the required quality bar, with fallback to a more capable model when necessary.

Microsoft’s current routing architecture similarly supports cost- and quality-oriented routing, sending simpler requests to smaller models and more demanding requests to larger ones.

Why this matters for enterprise AI cost

The economics become particularly important at scale.

Suppose an enterprise handles millions of requests every month.

If most requests are relatively simple, sending all of them to a large model can create unnecessary:

• token consumption
• compute
• memory usage
• latency
• GPU demand
• infrastructure cost

Using smaller models for suitable workloads can change those economics significantly.

This is one reason Gartner has been forecasting greater enterprise adoption of small, task-specific AI models, citing their potential for faster responses and lower computational and operational costs. (Gartner)

The right metric is still not:

“How cheap is this model?”

It is:

“How much does it cost to complete this business task reliably?”

That distinction matters because a cheaper model that fails frequently may cost more when you include retries, escalation and human intervention.

Smaller models can improve latency too

A smaller model can reduce more than cost.

It can reduce:

• inference time
• memory requirements
• queueing
• hardware pressure
• concurrency bottlenecks
• model transfer and loading overhead

Shunya’s published benchmarks report 3× faster inference for its evaluated Custom SLM workloads.

This becomes particularly important for high-volume customer-facing systems where small differences in latency can accumulate across thousands or millions of interactions.

For enterprise voice AI, the effect can be even more visible because customers experience every additional delay conversationally. Shunya’s current voice-agent page reports less than 1.5 seconds end-to-end voice-to-voice responseand describes a pipeline built from speech recognition, knowledge-grounded reasoning and speech generation.

Smaller models can change where AI can run

A smaller model does not just change cost.

It can change deployment options.

Large models often require substantial dedicated infrastructure.

Smaller models can make it easier to consider:

• private VPC
• on-premises
• CPU inference
• edge deployment
• offline operation
• embedded devices

Shunya’s on-device models provide a concrete example. Its Zero Tiny ONNX models are currently 5–15 MB, CPU-optimised and designed to operate without a GPU or internet connection.

That makes a different class of enterprise applications possible.

Instead of:

device → cloud → large model → response

some tasks can become:

device → local model → immediate response

with escalation to the broader intelligence stack when needed.

The most important question is task complexity

A common mistake is starting an AI project by asking:

“Which LLM should we use?”

A better process is:

Start with the task

What exactly should the AI do?

Define the quality bar

What does a correct answer or successful task look like?

Measure the workload

How frequently does the task occur? How much latency can it tolerate? How much variation exists in the inputs?

Identify the required knowledge

Does the model need broad world knowledge or specific enterprise context?

Decide what should be deterministic

Which parts should be handled by business rules, software or APIs?

Select the smallest model that meets the quality bar

Only after understanding the workload should the model be chosen.

Create an escalation path

When the smaller model is uncertain or the task is too complex, route it to a more capable model or a human.

This approach prevents enterprises from overbuilding the first version of the AI system.

What about accuracy?

This is usually the biggest concern when companies consider smaller models.

The right question is not:

“Is a smaller model as smart as a larger model?”

The right question is:

“Is the smaller model accurate enough for this specific task?”

A model that achieves 98% accuracy on a narrow classification task may be more useful than a much larger model that achieves 99% but costs significantly more and takes longer.

The acceptable threshold depends on the application.

For example:

a low-risk content categorisation task may tolerate a very different error rate from a financial workflow or clinical process.

That is why every model decision should be tied to a defined evaluation set.

Shunya’s benchmarks and production evaluation approach reflect this principle: model performance needs to be measured against the actual capabilities and workloads the system is expected to support.

A practical enterprise model strategy

For many organisations, a useful starting architecture is:

Tier 1: Deterministic systems

Use software, databases and business rules for tasks that should not require probabilistic reasoning.

Tier 2: Purpose-trained SLMs

Use smaller models for classification, extraction, routing, domain understanding and repetitive enterprise workflows.

Tier 3: Knowledge-grounded AI

Use knowledge graphs and RAG when the task needs current business knowledge or complex relationships.

Tier 4: Larger LLMs

Use larger models when the task genuinely requires broad reasoning, synthesis or open-ended generation.

Tier 5: Human escalation

Keep a human in the loop where the risk, uncertainty or business impact requires it.

This is not about replacing one model with another.

It is about assigning the right component to the right job.

You do not need an LLM for every enterprise AI application

This is the central idea.

A model that can write a 2,000-word research report is not necessarily the best model for identifying whether a customer wants to cancel an account.

A model that can reason across dozens of documents is not necessarily the best model for extracting an invoice number.

A model that can handle arbitrary questions is not automatically the best choice for a fixed workflow.

Enterprise AI should therefore move away from:

one model → everything

toward:

right model → right task

That can mean a combination of: custom SLMs, knowledge graphs, RAG, business rules, APIs, larger LLMs and human oversight.

Final thoughts

The enterprise AI question is changing.

It is no longer simply:

“Which LLM is the smartest?”

It is:

“Which architecture gives us the right intelligence for the task?”

For some workloads, the answer will still be a powerful large language model.

For many others, a purpose-trained SLM can be enough.

And for some tasks, neither is necessary because a database, API or deterministic rule can do the job more reliably.

The most efficient architecture is therefore not the one with the biggest model.

It is the one that uses:

• the right model
• the right knowledge
• the right context
• the right infrastructure
• and the right amount of computation

for every part of the workflow.

That is why enterprises do not necessarily need an LLM for everything.

They need an AI architecture that knows when an LLM is actually necessary.

Frequently Asked Questions

Does every enterprise AI application need an LLM?

No. Many enterprise applications contain tasks such as classification, extraction, routing, policy matching and workflow decisions that can be handled by smaller specialised models or deterministic software.

When should I use a Small Language Model?

An SLM is a strong candidate when the workload is narrow, repeatable, domain-specific, high-volume or latency-sensitive, and the model can meet the required accuracy threshold.

Are SLMs cheaper than LLMs?

They can be, particularly when their smaller computational requirements allow them to use less infrastructure and support more concurrent workloads. The actual economics depend on the model, hardware, deployment and workload.

Are smaller AI models faster?

Smaller models generally require less computation and memory, which can improve inference speed. The full application can still be slow if retrieval, APIs, network calls or other components dominate the latency.

Can an SLM replace an LLM?

For some workloads, yes. For others, no. A better architecture is often to use SLMs for specialised tasks and larger LLMs for complex reasoning, with routing between them.

What tasks can a small language model handle?

Common examples include intent classification, routing, entity extraction, structured outputs, domain terminology, workflow-state detection and repetitive enterprise tasks.

Do smaller models need less data?

Not necessarily. The amount and quality of training data depend on the task. A specialised model may need less broad-world knowledge because its scope is narrower, but its enterprise training and evaluation data still need to be high quality.

Can SLMs work with RAG and knowledge graphs?

Yes. A smaller model can use RAG and knowledge graphs to access enterprise information and relationships without needing to store all of that knowledge inside the model.

Can an SLM run on a device?

Some can. Shunya’s on-device models include CPU-optimised models designed for local and offline operation.

When should an enterprise still use a large LLM?

Use a larger LLM when the task genuinely requires broad reasoning, open-ended generation, complex synthesis, unpredictable inputs or multi-step analysis that a smaller specialised model cannot reliably handle.

Learn More

Explore Shunya’s Custom SLMs, enterprise AI platform, knowledge-grounded enterprise AI, hybrid Knowledge Graph and RAG architecture, enterprise voice agents, on-device AI models, and Shunya’s AI benchmarks.

Contact us to know more.