The Mixture-of-Models Era
The next model architecture will not fit inside one checkpoint.
vLLM Semantic Router is taking on a new mission: build Mixture-of-Models, a higher-level model architecture that turns a portfolio of independent models into one trainable, evaluable, and deployable model.
In less than a year, the project has grown to 5,000 stars, 150+ contributors, and more than 300,000 cumulative downloads across our Hugging Face model family. We shipped three major releases—Iris, Athena, and Themis—and watched a small experiment in semantic model selection turn into a serious open-source system for multi-model inference.
Those milestones close our first chapter. They also give us the technical foundation and the community to take on our next mission:
vLLM Semantic Router is becoming the training, evaluation, and inference engine for Mixture-of-Models.
Mixture-of-Models, or MoM, has been part of our system design from day 0. Until now, most of our public work has focused on the mechanism that makes it possible: routing. The next chapter moves the abstraction boundary higher. We want a complete model system—made from many models, policies, preferences, and execution paths—to be trained, evaluated, exported, imported, and run as one model.
If vLLM made a model fast and easy to serve, vLLM Semantic Router should make many models feel like one.
Figure 1: A Mixture-of-Models turns a heterogeneous model portfolio into one model experience.
What the First Chapter Proved
The first vLLM Semantic Router post started with a narrow, practical problem. Simple requests did not need the same reasoning budget as hard ones, so a lightweight classifier chose between fast and reasoning paths. The early router classified prompts into a fixed set of domains and helped vLLM spend inference compute more selectively.
That design was useful, but production traffic quickly broke its assumptions. A request is not defined by domain alone. It may also carry privacy constraints, safety risk, long context, language, modality, tool state, user preference, latency pressure, and authorization boundaries. A model endpoint can be cheap but overloaded, capable but remote, private but too small, or excellent for one turn and unsafe to switch into halfway through an agent session.
The architecture changed because the problem changed.
We first rebuilt the classifier layer around modular model support, shared LoRA computation, Rust/Candle inference, and Go integration. We then replaced fixed classification with a Signal–Decision architecture, separating observed evidence from policy and execution. That separation became the spine of the next three releases.
| Milestone | What changed |
|---|---|
| Initial release | Intent-aware selection between fast and reasoning paths |
| v0.1 Iris | Signals, decisions, and route-scoped plugins replaced fixed classification |
| v0.2 Athena | Model selection, memory, RAG, long context, multimodality, and a new router model stack formed a system brain |
| v0.3 Themis | Stateful routing, projections, replay, protocol support, session continuity, and one production configuration contract made the system operable |
| Fusion and Micro-Agent | The router began choosing collaboration patterns, not only individual models |
Figure 2: Each stage changed the unit of control: model, decision, system, session, and finally the complete model lifecycle.
Iris made routing composable. Signals for domain, keywords, embeddings, factuality, feedback, and preference fed explicit decisions; safety, PII protection, semantic caching, hallucination detection, and tool selection became route-scoped behavior. Iris also introduced the MoM model family and described vLLM-SR as System Level Intelligence for Mixture-of-Models.
Athena expanded the scope. It refreshed the multilingual and multimodal model stack, made model-selection algorithms first-class, brought memory and RAG into the runtime, accelerated long-context classifiers on AMD ROCm, and turned the dashboard into an operating surface. The project was no longer a classifier in front of vLLM. It was becoming a system brain.
Themis gave that brain a durable contract:
Signals become projections. Projections feed decisions. Decisions choose algorithms. Algorithms select models.
Themis added session-aware agentic routing, replayable traces, stronger OpenAI and Anthropic protocol support, an operator console, and broader runtime paths across AMD ROCm, NVIDIA CUDA, Intel OpenVINO, and CPU environments. It made an important promise to operators: a route should be explainable. You should be able to see what evidence fired, which policy matched, which algorithm ran, which physical model served the request, and why.
Two papers made the system argument explicit
The releases built the runtime. Two project papers explained the architecture behind it.
The white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalized the separation between neural evidence and symbolic policy. It describes routing in two regimes. In the first, signal extraction reduces uncertainty about which model or path fits a request. Fast heuristics and learned classifiers turn raw prompts, context, identity, safety, and modality into a structured signal vector. In the second, a Boolean decision engine composes those signals into an auditable policy. At the time of the paper, the system covered thirteen signal types and thirteen model-selection algorithms, with per-decision plugin chains for caching, RAG, memory, safety, provider handling, and response validation.
The typed neural-symbolic DSL is an important part of that design. It acts as the instruction set for the routing engine: parse policy into a Boolean expression tree, validate syntax and references, and compile the same intent into deployable configuration. Neural models remain good at extracting uncertain evidence; symbolic policy remains reviewable, testable, and suitable for infrastructure control.
The vision paper, The Workload–Router–Pool Architecture for LLM Inference Optimization, widened the frame. It argues that three variables have to be designed together:
- Workload: chat or agent, single-turn or multi-turn, warm or cold, prefill-heavy or decode-heavy
- Router: static semantic policy, online feedback or bandit adaptation, RL-based selection, and quality-aware cascades
- Pool: homogeneous or heterogeneous accelerators, prefill/decode topology, model placement, and KV-cache management
The coupling is the contribution. Workload shape changes which routing policy works; routing policy changes the required pool size and topology; pool state changes which route is efficient. Safety and privacy cut across all three dimensions, while cost, quality, latency, and energy define the optimization frontier. The paper maps the project’s research into a 3 × 3 WRP matrix and identifies twenty-one open directions where those dimensions still need to meet.
Figure 3: The white paper defines the programmable routing engine; the vision paper connects it to workload and physical pool design.
Together, the papers established the two halves of the next chapter. The white paper explains how to make a routing decision programmable. The vision paper explains why that decision cannot be separated from the workload and the machines that execute it. Mixture-of-Models brings both into one model abstraction.
Then the unit of execution changed again. Fusion, ReMoM, Confidence, Ratings, and bounded Workflows let one request invoke a controlled collaboration among models. As the Micro-Agent work showed, a client can call one model name while the serving layer selects a recipe, fans out to workers, verifies or synthesizes their results, and returns one ordinary response.
That is the line between the two chapters.
| First chapter | New chapter |
|---|---|
| Route a request | Build a model system |
| Choose a model or capability path | Train, evaluate, and execute the whole MoM |
| Configure runtime policy | Package a portable, versioned model artifact |
| Optimize a routing decision | Optimize system intelligence across quality, cost, latency, safety, and energy |
| Hide backend choice behind one API | Make the complete multi-model system behave like one model |
Routing remains fundamental. It is how a Mixture-of-Models allocates work, applies policy, and coordinates its parts. But routing is the mechanism. The model system is the product.
Why We Need a Higher-Level Model Abstraction
The “one best model” era is not coming. AI is settling into a fragmented but productive equilibrium.
Models are fragmented. Closed frontier models, open general models, domain experts, compact local models, verifiers, embeddings, and multimodal models will coexist. None wins across quality, cost, latency, trust, privacy, modality, and domain fit at the same time.
Compute is fragmented. New and old GPUs, CPUs, specialized accelerators, edge devices, cloud capacity, and private clusters have different memory limits, kernels, availability, prices, and energy curves. Model choice and compute placement are becoming the same decision.
Location is fragmented. Inference now spans cloud, data center, and edge. Some data must stay local. Some workloads benefit from an on-demand cloud expert. Others must remain inside a regulated boundary even when a remote model would score better on a benchmark.
Preference is fragmented. There is no universal “best.” Products and users make different tradeoffs among accuracy, latency, price, privacy, hallucination tolerance, safety, style, and multimodality. Those preferences should shape execution directly.
The durable bottleneck is therefore not model access. It is intelligent allocation: deciding which intelligence should run, where it should run, how much of it to spend, and whether several models should work together.
This is also an energy problem. Hardware and inference engines improve the supply side by producing more tokens per watt per dollar. MoM addresses the demand side: which semantic work deserves those tokens at all? Sending every request to the largest model wastes capacity; forcing every request onto the cheapest model wastes capability. A mature system has to know the difference.
When that logic lives independently in every application, it spreads across prompts, SDK wrappers, gateway rules, and agent graphs. It becomes difficult to reproduce, govern, or improve. A higher-level model abstraction gives that logic one home.
Figure 4: Fragmentation turns model routing into an allocation problem across intelligence, placement, and budget.
What We Mean by Mixture-of-Models
A Mixture-of-Models is a versioned composite model whose engine realizes each request through a preference-conditioned, resource-bounded path across independent models and operators. It is presented to the user through one model interface and returns one attributable result.
MoM is not another name for a gateway with several upstreams. A gateway can forward traffic without owning the quality of the resulting system. An MoM has an objective, an evaluation contract, a reproducible composition, and a runtime responsible for executing that composition.
MoM is also different from Mixture-of-Experts. MoE routes tokens among internal experts during one model’s forward pass. MoM coordinates independent models at the system level. Its components may use different architectures, owners, licenses, modalities, protocols, context windows, and hardware. An MoE checkpoint can itself be one component inside an MoM.
| Conventional model | Mixture-of-Models | |
|---|---|---|
| Unit of intelligence | One checkpoint | A governed system of models |
| Specialization | Primarily encoded in weights | Composed across independent specialists |
| Execution | One generation path | Selection, cascade, verification, fusion, or workflow |
| Optimization target | One model’s quality and efficiency | The system frontier across quality, cost, latency, safety, privacy, and energy |
| Deployment boundary | One runtime | Cloud, data center, and edge |
| User contract | One model identity | One model identity |
Figure 5: Selection is one MoM topology. Cascades, parallel fusion, and bounded workflows share the same model boundary.
This changes what a model artifact needs to contain. Weights and configuration are still part of the story, but a portable MoM also needs a component manifest, capability metadata, routing and collaboration recipes, policies, preferences, evaluation suites, runtime constraints, provenance, and version history.
Open checkpoints can travel with the artifact. Closed models can be preserved as authenticated external references with explicit capability and policy contracts. Exporting an MoM does not make a proprietary checkpoint portable; it makes the model system reproducible.
Preference as a model contract
This abstraction becomes useful when it is as concrete as a model name. The draft Mixture-of-Models Bundle proposal and position paper illustrate one family with several published operating points:
| Model identity | Contract |
|---|---|
vllm-sr/mom-v1-flash |
Minimize expected latency |
vllm-sr/mom-v1-light |
Minimize cost above a quality floor |
vllm-sr/mom-v1-ultra |
Maximize quality within a declared budget |
vllm-sr/mom-v1-halu |
Require grounding checks and fail-closed fallback |
vllm-sr/mom-v1-secu |
Enforce jailbreak and PII policy before execution |
The point is not the suffix. It is that preference becomes part of the model contract. flash is trained and evaluated on a latency–quality frontier; ultra can spend a declared budget to pursue higher quality; halu and secu make grounding and security gates part of the model itself. When that family evolves, mom-v2-ultra can introduce a new policy or topology without silently changing mom-v1-ultra. The application chooses a model. vLLM-SR owns the complexity behind it.
A request can still tune a published tradeoff within the model’s declared range. For example, ultra may accept a small latency bias while remaining accuracy-first. It cannot quietly become light while keeping the ultra identity. And no soft preference may override a privacy, residency, authorization, or safety constraint. The engine decides what is admissible before it decides what is optimal.
To an application, the full system remains an ordinary model call:
{
"model": "vllm-sr/mom-v1-ultra",
"messages": [
{"role": "user", "content": "Review this design and identify its weakest assumption."}
]
}
Internally, that identity may choose one model, escalate through a cascade, compare parallel answers, require a grounding check, or run a bounded workflow. Externally, it keeps one interface, one version, and one response.
Figure 6: Preferences are published as bounded, versioned model contracts—not hidden application-side routing presets.
The architecture we are building has four planes with clear ownership:
| Plane | What it owns | Foundation already in vLLM-SR | Next step |
|---|---|---|---|
| Artifact | Components, capabilities, objectives, policy, eval contract, provenance | Canonical config, model references, DSL, versioned policy | Portable MoM import/export specification |
| Learning | Router-owned models, preferences, outcomes, recipe improvement | Training stack, Router Learning, replay, outcome APIs | Joint training and system-level release gates |
| Execution | Signals, projections, decisions, selectors, loopers, plugins | Signal–Decision runtime, Fusion, ReMoM, Workflows, safety and memory | One lifecycle-aware MoM engine |
| Physical | Providers, model pools, accelerators, locality, cache and energy state | vLLM backends, cloud providers, ROCm, CUDA, OpenVINO, CPU | Portable placement across cloud, data center, edge, and local devices |
Figure 7: A complete MoM spans four planes: artifact, learning, execution, and physical realization.
The portable artifact is only the first boundary. A deployment has to map logical requirements onto the models and machines available in one environment. The proposal separates that lifecycle into four objects:
- The bundle fixes the model interface, graph, policies, contracts, behavior variant, bounds, and immutable semantic assets.
- The binding maps logical components to eligible deployments without changing the model’s decision semantics.
- The resolution lock freezes the exact constituent revisions, runtimes, images, accelerators, and provider observations used for a realization.
- The run record attributes every decision, call, constraint check, cost, and outcome to the bundle, binding, and lock that produced it.
This separation is what makes portability honest. The same mom-v1-ultra can bind to a ROCm cluster, a CUDA fleet, a private CPU or NPU edge node, or a hybrid of local and managed models while preserving its logical contract. It does not promise identical outputs from opaque providers. It preserves the model’s control semantics, makes substitutions explicit, and gives serving and evaluation the same resolved system to execute.
vLLM-SR as the MoM Engine
Training, evaluation, and inference have to share one system contract. Otherwise routing research stays in notebooks, evaluation measures a different system from the one deployed, and production behavior drifts away from both.
Training allocation, not only weights
MoM training includes the models owned by the router—embeddings, signal encoders, preference models, safety models, and selectors—but it goes further. The system must learn which model or collaboration pattern performs best for a workload under a given budget and environment. It must learn stopping rules for cascades, judge and synthesis policies for panels, session-switch economics for agents, and preference-aware tradeoffs. Because constituent models may be independent or closed, progress does not require gradients through every model: policies, thresholds, model pools, prompts, contracts, and execution topology can all be optimized from traces and outcomes.
The optimization target is a frontier, not one accuracy number. Quality, latency, cost, safety, privacy, reliability, locality, and energy all matter. Replay and outcome data connect production experience back to offline training without allowing the hot path to rewrite policy silently.
Evaluating the MoM as one model
If users invoke one model identity, evaluation should score that identity end to end. Backend benchmark scores are inputs, not the final answer.
MoM evaluation needs to measure routing regret, collaboration gain, failure recovery, session continuity, tail latency, cost, privacy and safety outcomes, and tokens per watt. It should test provider failures, device loss, model disagreement, workload drift, and preference changes. Just as importantly, it must evaluate the declared operating point: flash on its latency–quality frontier, light against its quality floor, and ultra within its resource budget. The result should be a versioned scorecard that serves both as training evidence and as a release gate.
The scientific test is stricter than asking whether more model calls improve a benchmark. Under matched active compute, can a conditional system exploit complementary model strengths and failure modes better than the best fixed model? Without that control, MoM can hide brute-force scaling behind a clever graph. Our evaluations must therefore report active calls, tokens, cost, latency, and energy alongside quality—and publish the conditions where composition does not help.
Executing intelligence at inference time
At inference time, the engine decides whether one model is enough. It may choose a local specialist, preserve a warm session on the current model, escalate through a confidence cascade, require retrieval or verification, run a Fusion panel, or execute a bounded workflow. The runtime owns the budget, topology, fallback policy, trace, and response contract.
The application should not have to reconstruct that machinery. It should make a normal model call.
Figure 8: MoM is a closed lifecycle: train the allocation policy, evaluate the full system, execute it, and turn outcomes into the next validated version.
One Model That Can Move
Our target is a complete MoM that can be built, exported, imported, versioned, evaluated, deployed, and invoked as a unified model. The proposal deliberately puts that model lifecycle above today’s runtime configuration: author a logical specification, compile it into a typed intermediate representation, package an immutable bundle, bind it to an environment, resolve and lock the concrete deployment, then use that same identity for serving and evaluation.
The same artifact should run across a developer machine, a private cluster, a cloud fleet, and an edge environment while preserving its model identity and behavioral contract. Its physical realization can change with the environment: a logical specialist may resolve to an admissible local checkpoint or managed endpoint, and one accelerator runtime may be replaced by another. If a remote expert is unavailable behind a privacy boundary, the engine must follow a fallback or abstention path already declared by the model. The binding may not silently rewrite the graph, relax a guard, or turn a panel into a cascade; those changes require a new model version.
“Run on any hardware” is an architectural requirement, not a claim that every component is portable today. The project already has concrete paths across ROCm, CUDA, OpenVINO, and CPU execution. The next step is to make hardware capability and placement part of the MoM contract, so the engine can map the same model system onto what is actually available.
The standard for the user experience is simple:
One model identity. Many models. Any hardware.
If the application needs to know which provider owns every submodel, which device runs it, or which fallback graph to execute, the abstraction has leaked.
What Changes Now
The next stage of vLLM Semantic Router will concentrate on four connected areas:
- Define a portable MoM specification. Represent components, capabilities, objectives, policies, preferences, evaluation contracts, runtime constraints, and finite execution semantics as one versioned artifact.
- Close the training–evaluation–inference loop. Use evaluation and replay to improve router-owned models and recipes, validate changes against the system frontier, and deploy them through reviewable and rollback-safe releases.
- Build a heterogeneous runtime. Map the same MoM across cloud, data center, and edge, using hardware, locality, energy, and data boundaries as execution inputs.
- Keep the model interface boring. Import, export, deploy, and invoke an MoM with the same ease expected from a single model. Complexity belongs inside the engine.
This is larger than adding more routing algorithms. It asks how independent models should specialize, compete, verify, and collaborate; how system intelligence should be measured; and how one model contract can survive across devices and environments.
That leads to our new mission:
Advancing the science of intelligence across models, devices, and environments.
Across models means learning how a portfolio can produce capability beyond any one checkpoint. Across devices means treating placement, availability, and energy as part of intelligence. Across environments means carrying the same model experience from edge to cloud and from research to production.
Build It With Us
Five thousand stars is not a finish line. It is evidence that this problem matters to more people than the original team.
Thank you to everyone who trained a model, opened an issue, reviewed a pull request, added a backend, tested a release, wrote documentation, shared a benchmark, or forced us to revise an assumption. Iris, Athena, and Themis were built by that community.
The next chapter needs model researchers, evaluation researchers, systems engineers, hardware teams, model builders, and operators with real workloads. We want collaborators working on learned allocation, preference optimization, model cooperation, energy-aware inference, portable artifacts, open evaluation, and heterogeneous runtimes.
Join us on GitHub, explore the documentation, try the MoM model family, and meet the community in the #semantic-router channel on vLLM Slack.
The first chapter taught infrastructure how to choose a model.
The next chapter is about building the model that emerges when models, devices, and environments learn to act as one.
Welcome to the Mixture-of-Models era.