mtlm-router / vs llm function-calling
The common way to turn natural language into a route is prompt a big model and parse JSON or a tool call. It works — and for some teams it's the right answer. This page is the honest trade-off, with our numbers where we have them and gaps marked where we don't.
| mtlm-router | LLM function-calling | |
| latency | p50 ~180ms on CPU (measured, disclosed) | ~1–5s typical; TTFT varies by load |
| marginal cost / call | ~€0 — runs on your CPU | per-token; a 51-way route costs the full prompt + output every time |
| data leaves your network? | no — single binary, air-gap OK | yes, unless you self-host a large model |
| "I don't know" behavior | calibrated: conformal prediction sets, per-lane qhat — abstains with measured coverage, returns candidates to clarify | prompted: "say oos if unsure" — uncalibrated, drifts silently, refuses or guesses per model mood |
| changing the taxonomy | retrain a ~KB head (hours, CPU); trunk stays frozen | edit a prompt (minutes) — but re-validating behavior is on you |
| eval evidence | per-lane eval.json + re-runnable samples — claims you can falsify | your own harness; provider model upgrades can shift results under you |
| open-domain knowledge | none — it routes, it doesn't know things | strong — this is what LLMs are for |
| day-one coverage of a new domain | needs a few dozen phrasings per route | zero-shot — works immediately, quality unmeasured |
| accuracy ceiling | 73–99% by lane width — honest numbers per lane, some lanes triage-only | high on easy taxonomies, but opaque error modes on boundary cases |
The router isn't competing with your LLM — it sits in front of it. Every route decision can delegate upstream (ANVIL_DELEGATE_URL): confident in-scope → cheap tool call; uncertain/out-of-scope → your LLM, where a big model's knowledge is genuinely needed. You pay LLM prices only for the fraction of traffic that deserves it — typically the clarification tail, not the routed bulk.