Routing to the ideal AI agent to save compute
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.
Nvidia has released NeMo Switchyard, an open source model router that sits between an application and a pool of language models and decides, per request or per turn, which model should handle a particular task.
The aim is to maximize efficiency by picking the right model for the right task: pushing frontier-level models to do simple tasks that much smaller or cheaper models could handle is not only inefficient, but also costly for enterprise customers.
This comes at a time when enterprise customers are already grappling with rising costs that can quickly spiral out of control as they increasingly adopt AI.
The problem Nvidia is trying to solve isn't new, and industry giants are already attracting a lot of attention. The emerging field itself is potentially lucrative, even if it focuses on cutting costs, and major players are looking to get in on the action; Stripe's upcoming $7+ billion acquisition of OpenRouter, underscores this.
Switchyard is no different from its peers; it is essentially a proxy that uses a routing algorithm to decide which AI model to send queries to, while delivering an answer in the format that the calling application or user requires.
It accepts OpenAI, Anthropic, and Responses API requests, translates between them, and documents the selected model, decision rationale, token usage, and latency for each call, enabling transparency and letting it tweak or tune its approach over time.
This makes sense compared with a one-size-fits-all approach that would otherwise be prevalent in a world with limited frontier-level AI compute, as LLMs continue to grow larger and more demanding. One example is what Nvidia points to: Nemotron Parse, a one-billion-parameter model primarily designed to extract structure from PDFs.
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
The question remains, however, whether such an approach is always feasible. Leveraging such a routing tool comes at a cost; LangChain's test of Nemotron 3.5 Lightning found that its judge model, which determines whether the agent is still on track after each turn, consumed up to 21.2% of total cost, second only to what it spent on Claude's Opus 4.8.
Running a more optimized judge model could therefore yield greater savings or lower overhead, but it could also hinder cases where users need a more basic AI model, and the judge model increases costs by reading or checking output at every turn.
A secondary concern would be variance: while costs are noticeably lower than running just Opus 4.8 in testing, it does swing from anywhere between $2.16 and $3.61, a mammoth 67% movement essentially guided by when it escalated queries to a bigger or smarter model. This tradeoff makes cost hard to predict: while routing lowers average spend, it widens the distribution around it.
LangChain also says users should not use Switchyard if their workloads are short or latency-sensitive; it estimates it adds about 700ms of latency because the judge model has to read all output.
Nvidia's take here is simple: it is pushing a packaged, open-source solution that offers a broad set of integrations in what is essentially a relatively fragmented space. Routing work to smaller open-weight models pushes inference toward hardware enterprises own, which Nvidia sells.
