Routing research

Reframes routing as deliberate reasoning: the router jointly picks the model AND its output-length budget via length-constrained instructions, instead of only picking a model.

A deliberately simple production router: a LinUCB contextual bandit over lexical + sentence-embedding features, with an offline reward matrix built by scoring every candidate model on a curated prompt set.

Routes on internal LLM activations (the residual stream captured at prefill) rather than semantic embeddings, with “Encoder-Target Decoupling” so one shared trunk can score every model in the pool.

Treats a whole session, not a single query, as the routing unit — a finite-horizon MDP with remaining budget in the state, solved with Conservative Q-Learning so the policy learns delayed gratification.

Systematic survey of multi-LLM routing and cascading — routing ACROSS independently trained models, explicitly contrasted with mixture-of-experts routing inside one model — under a when / what / how taxonomy.

Consolidates the vLLM Semantic Router project into three co-equal axes: Workload (chat vs agentic, prefill- vs decode-heavy), Router (semantic rules, bandits, RL, quality-aware) and Pool (GPU topology).

Replaces a cascade's confidence threshold with conformal prediction set size: accept the small model's answer when the calibrated set has one answer, otherwise escalate. This gives a distribution-free, finite-sample accuracy bound (at least 1 - K*alpha) and states expected cost as a function of alpha. It needs no training, only black-box API access. On 18 multiple-choice benchmarks with two-tier cascades from four open-weight families, it beats the strongest calibration-tuned heuristic cascade on most family-benchmark pairs.

Counts the upfront cost of collecting query-model quality labels (running every candidate on past queries) against the serving savings a router earns. Proposes SaveRouter, which acquires only informative feedback and shares capability information across related queries. Routing is scored on break-even deployment volume, not only on serving cost. Code is at github.com/LAMDA-Model-Reuse/SaveRouter.

A benchmark for per-step agentic routing, where the router picks a model at every agent step from a locked pool. Static track: 970 router-visible prefixes from 520 instances across SWE-bench, BFCL, mtRAG, QMSum and PinchBench.

Tests six commercial routers in 14 settings across eight task categories. The developers include OpenRouter, Microsoft Azure, vLLM, Not Diamond, Nadir and Orca. None beats a router that picks randomly between two well-chosen models at matched cost, and some trail it by more than 10 points. The paper names four failure patterns: difficulty blindness, length reversal, semantic matching and roster suboptimality. It also proposes an evaluation method that does not reward them.

Compares 21 routing methods on five benchmarks under one setup. Most of them, including kNN, land in a narrow accuracy band well below the per-query oracle. The authors argue the bottleneck is predicting correctness: routers learn each model's average strength, not query-specific signals. On a new 300K-query benchmark (Nine-by-300k), more data, bigger encoders and fine-tuning lift eight routers by only 1.24 pp on average.

Replaces autoregressive drafting with a lightweight block diffusion model that generates a whole block of draft tokens in one parallel forward pass, conditioned on target-model context features.

Splits prefill from decode across datacenters: long-context prefill is offloaded to compute-dense clusters and the resulting KVCache shipped over commodity Ethernet — not an RDMA fabric — to the decode site.

When a prompt's nearest curated-answer neighbour falls just below the similarity threshold, Krites asynchronously verifies whether the curated answer would in fact have served — and promotes it — instead of forcing one static threshold to decide.

Preview of the V4 series: two open-weight MoE models (Pro 1.6T/49B active, Flash 284B/13B) at 1M context, with hybrid compressed sparse attention.

The K3 technical report: 2.78T total / 104.2B activated MoE with native vision and 1M context, built on Kimi Delta Attention plus Attention Residuals, interleaving KDA with Gated MLA 3:1.

Shows that stateless provider failover keeps the service up but silently drops conversation history. Proposes a proxy that forwards that history to the fallback provider. Defines two metrics, Continuity Preservation Rate and Continuity Latency Overhead, and releases an open harness (continuity-bench). It also finds that fixed-interval retries against rate-limited fallbacks cause retry storms, and that exponential backoff with jitter fixes them.

Ablates four context configurations on a 50-task expense-itemization benchmark in Dynamics 365 via MCP tools, 5 runs each. Keeping only the last five tool calls plus a summary beat passing full history.

Moves agent memory out of the prompt and into weights — experience is distilled into fast LoRA updates online, so behaviour changes within a session without growing the context.