Routing research

Reframes routing as deliberate reasoning: the router jointly picks the model AND its output-length budget via length-constrained instructions, instead of only picking a model.

A deliberately simple production router: a LinUCB contextual bandit over lexical + sentence-embedding features, with an offline reward matrix built by scoring every candidate model on a curated prompt set.

Routes on internal LLM activations (the residual stream captured at prefill) rather than semantic embeddings, with “Encoder-Target Decoupling” so one shared trunk can score every model in the pool.

Treats a whole session, not a single query, as the routing unit — a finite-horizon MDP with remaining budget in the state, solved with Conservative Q-Learning so the policy learns delayed gratification.

Systematic survey of multi-LLM routing and cascading — routing ACROSS independently trained models, explicitly contrasted with mixture-of-experts routing inside one model — under a when / what / how taxonomy.

Consolidates the vLLM Semantic Router project into three co-equal axes: Workload (chat vs agentic, prefill- vs decode-heavy), Router (semantic rules, bandits, RL, quality-aware) and Pool (GPU topology).

A benchmark for per-step agentic routing, where the router picks a model at every agent step from a locked pool. Static track: 970 router-visible prefixes from 520 instances across SWE-bench, BFCL, mtRAG, QMSum and PinchBench.

Replaces autoregressive drafting with a lightweight block diffusion model that generates a whole block of draft tokens in one parallel forward pass, conditioned on target-model context features.

Splits prefill from decode across datacenters: long-context prefill is offloaded to compute-dense clusters and the resulting KVCache shipped over commodity Ethernet — not an RDMA fabric — to the decode site.

When a prompt's nearest curated-answer neighbour falls just below the similarity threshold, Krites asynchronously verifies whether the curated answer would in fact have served — and promotes it — instead of forcing one static threshold to decide.

Preview of the V4 series: two open-weight MoE models (Pro 1.6T/49B active, Flash 284B/13B) at 1M context, with hybrid compressed sparse attention.

The K3 technical report: 2.78T total / 104.2B activated MoE with native vision and 1M context, built on Kimi Delta Attention plus Attention Residuals, interleaving KDA with Gated MLA 3:1.

Ablates four context configurations on a 50-task expense-itemization benchmark in Dynamics 365 via MCP tools, 5 runs each. Keeping only the last five tool calls plus a summary beat passing full history.

Moves agent memory out of the prompt and into weights — experience is distilled into fast LoRA updates online, so behaviour changes within a session without growing the context.