Model architectures

The 2017 seq2seq Transformer: a bidirectional encoder reads the source, a causal decoder writes the target via cross-attention.

The encoder stack applies full bidirectional self-attention over the input sequence to produce contextual representations in one parallel pass. The decoder stack autoregressively generates the output: each decoder layer runs causal (masked) self-attention over tokens generated so far, then cross-attention where decoder queries attend over the encoder's output keys/values, then a feed-forward block. Sinusoidal positional encodings inject order; multi-head attention lets different heads specialize. This replaced recurrence entirely, making training parallelizable across sequence positions and unlocking the scaling that all later LLMs build on.

Bidirectional encoder pretrained by masking tokens; still the workhorse for embeddings, rerankers, and classifiers, revived by ModernBERT.

Take only the Transformer's encoder stack, so every token attends to every other token in both directions, and pretrain with masked language modeling: randomly mask ~15% of tokens and predict them from full bidirectional context. The result is a deep contextual representation of the whole input in a single parallel forward pass — no autoregressive loop. Downstream you attach a small head (a classifier on [CLS], a token tagger, or a pooling layer for embeddings) and fine-tune. RoBERTa showed BERT was undertrained (more data/steps, dynamic masking, drop NSP); DeBERTa added disentangled content/position attention; ModernBERT (2024) retrofitted the recipe with RoPE, GeGLU, alternating local/global attention, Flash Attention, 8k context, and 2T training tokens.

T5/BART framed every NLP task as text-in, text-out with a seq2seq Transformer; Flan-T5, UL2, and 2025's T5Gemma keep the line alive.

Keep both Transformer stacks but pretrain on denoising: T5 masks contiguous spans and trains the decoder to emit the missing spans (span corruption), casting every task — translation, summarization, QA, classification — as 'text in, text out' with a task prefix. BART instead corrupts whole documents (deletion, shuffling, infilling) and reconstructs them, which transfers especially well to summarization. UL2 unified the objectives as mixture-of-denoisers (short spans, long spans, prefix-LM) switchable by a mode token. The 2025 revival, T5Gemma (paper title 'Encoder-Decoder Gemma'), adapts pretrained decoder-only Gemma 2 checkpoints into encoder-decoder form via UL2/prefix-LM continued pretraining, showing better quality-per-inference-FLOP than the source decoder-only models on input-heavy tasks.

The 2018-2020 OpenAI recipe: causal decoder stack with learned absolute positions, LayerNorm, GELU MLPs, standard multi-head attention.

Token IDs are embedded via a BPE vocabulary and summed with a learned absolute position embedding table, then pushed through N identical blocks: causally-masked multi-head self-attention followed by a 4x-width two-layer GELU MLP, each wrapped in a residual connection with LayerNorm. GPT-1 placed LayerNorm after each sublayer (post-norm, as in the 2017 Transformer decoder minus cross-attention); GPT-2 moved it before each sublayer and added a final LayerNorm, which is what made 48+ layer training stable. A final LayerNorm feeds an unembedding projection (weight-tied to the input embedding) producing next-token logits; the entire training objective is next-token prediction. GPT-3 kept this recipe at 175B, alternating dense and locally-banded sparse attention layers.

The post-2023 dense standard: pre-RMSNorm, rotary positions (RoPE), SwiGLU gated MLPs, grouped-query attention, no bias terms.

Keeps the GPT causal-decoder skeleton but swaps every component: RMSNorm (rescale by root-mean-square only, no mean-centering or bias) is applied before each sublayer; positions are injected inside attention by rotating query/key vectors with position-dependent angles (RoPE), giving relative encoding and enabling long-context extension via theta rescaling. The MLP becomes SwiGLU — SiLU(gate_proj(x)) * up_proj(x) fed to down_proj, with hidden width ~8/3·d to hold parameter count. Attention uses grouped-query attention: many query heads share a small set of K/V heads, shrinking the KV cache several-fold with near-MHA quality; linear layers drop bias terms. Llama 1 established pre-RMSNorm+RoPE+SwiGLU (still MHA); Llama 2 70B and Mistral 7B added GQA; Llama 3 applied GQA at every size with a 128K vocab and RoPE theta 500k.

Three frontier labs replaced the transformer's single residual stream with four parallel ones, mixed by a routing matrix that Sinkhorn iteration constrains to stay close to the identity.

Since 2017 every block has read from and written back to one residual stream. Hyper-connections widen it: run N streams in parallel and learn how each block's input is drawn from them and how its output is distributed back. The danger is that a freely-learned mixing matrix destroys the identity path that makes deep networks trainable at all, so mHC — Manifold-Constrained Hyper-Connections — projects the mixing matrices onto the Birkhoff polytope (the set of doubly-stochastic matrices) using Sinkhorn-Knopp iteration, which alternately normalizes rows and columns until both sum to one. A doubly-stochastic mix redistributes signal between streams without amplifying or attenuating it, which is what keeps the residual path well-behaved. The shipped configs are unusually legible about this. GLM-5.3-Flash sets mhc: true, hc_mult: 4 (four streams) and hc_sinkhorn_iters: 20 with hc_eps 1e-6. Qwen3.8-Flash-Next carries hc_count: 4 with hc_lowrank: 320 against a hidden size of 2,560 — the same four streams, with the mixing factored low-rank. Tencent's Hy4-preview ships a variant it calls iHC (identity Hyper-Connections) with enable_ihc: true, hc_mult: 4 and hc_magnitude: 2.0. Three independent labs, three codebases, all at exactly four streams. Note the literature has already moved past what GLM shipped: mHC-lite is titled, in so many words, "You Don't Need 20 Sinkhorn-Knopp Iterations" — and 20 is exactly what the GLM-5.3-Flash checkpoint runs.

The residual stream adds every previous sublayer's output with weight one. AttnRes replaces that sum with softmax attention over preceding layers, so each layer chooses how much of each earlier layer to read.

In a PreNorm transformer the residual stream is a running sum: layer n sees everything before it, each contribution weighted identically at 1.0. The Kimi team's argument is that this is doing real damage at depth — uniform accumulation causes uncontrolled hidden-state growth and progressively dilutes any single layer's contribution, since one new output is added to an ever-larger sum. AttnRes replaces the accumulation with softmax attention taken over the outputs of preceding layers, so aggregation along the depth axis becomes learned and input-dependent rather than fixed. Attending over every preceding layer output is expensive in memory and cross-stage communication, so the shipped form is Block AttnRes: layers are partitioned into blocks and the attention runs over block-level representations, with cache-based pipeline communication and a two-phase computation strategy. The reported effect is more uniform output magnitudes and a better-distributed gradient across depth. This entry exists because the atlas already had the evidence and not the name: the Kimi K3 deep dive recorded `attn_res_block_size: 12` straight out of the released config without being able to say what it was. It is this — Block AttnRes with block size 12. The check that makes it solid is the negative control: Kimi K2.5's config carries no attn_res field at all, so this is K3-specific machinery rather than something inherited across the family.

Attention sinks and huge activation outliers look like pathologies. The argument here is that they are the model improvising a rescaling mechanism it was never given — so give it one, and the outliers stop being necessary.

Two long-standing oddities of trained transformers are attention sinks (a handful of tokens absorbing enormous attention mass) and residual sinks (a few dimensions carrying persistently huge activations). Both are usually treated as artifacts to be worked around, and both are exactly what makes low-bit quantization hard, because one outlier dimension sets the scale for everything. This paper's claim is that they are functional: together with softmax and RMSNorm they act as an implicit rescaling of the non-outlier components, so the outliers serve as rescale factors rather than as contributors. That reframing yields a prescription. If the network is growing outliers because it needs a rescaling knob, hand it one explicitly, and it will stop. GatedNorm does that — a lightweight elementwise low-rank self-gate applied after normalization, GatedNorm(u) = RMSNorm(u) ⊙ σ(W2·SiLU(W1·RMSNorm(u))). The reported evidence is nicely symmetrical: removing normalization kills the outliers but hurts stability, and clipping the outliers while keeping normalization also degrades — but absorbing the rescaling into learnable parameters recovers about two points on average and, tellingly, makes W4A4 quantization far more robust (1.2 points of degradation). That last number is the practical point: outliers are the reason 4-bit activations are hard, so a model that no longer needs them quantizes better by construction.

Share key/value heads across query heads — same attention math, 8-64x smaller KV cache for fast batched decoding.

In multi-head attention (MHA) every one of the h query heads has its own K and V head, so the decode-time KV cache costs 2 x n_layers x n_kv_heads x d_head x bytes per token — for a Llama-2-70B-shaped model (80 layers, 64 heads, d_head 128, fp16) that is ~2.6 MB per token, which dominates GPU memory and bandwidth at long context and large batch. MQA (Shazeer 2019) keeps h query heads but a single shared K/V head, cutting the cache by h-fold (~41 KB/token in the same config) at some quality cost. GQA interpolates: query heads are partitioned into g groups, each sharing one KV head (Llama 2 70B uses g=8, ~0.33 MB/token), recovering near-MHA quality; the GQA paper also showed an existing MHA checkpoint can be converted by mean-pooling its KV heads and uptraining with ~5% of original compute.

Compress each token's K/V into one small latent vector, reconstruct full heads on the fly — ~93% KV-cache cut without losing head diversity.

Instead of caching per-head keys and values, a learned down-projection compresses the token's hidden state into a single latent c_KV (DeepSeek-V2: 512 dims) plus a small decoupled RoPE key (64 dims) — only ~576 floats per token per layer are cached, versus 2 x 128 heads x 128 dims = 32,768 for equivalent MHA (~57x smaller per layer; DeepSeek reports 93.3% total KV reduction, comparable to GQA with just 2.25 groups). At attention time, up-projection matrices reconstruct all 128 per-head K/V — and because these are linear, they can be absorbed into the query and output projections so inference attends over the latent directly. The RoPE path must be decoupled because rotary position rotation does not commute with the low-rank compression.

Each token attends only to a fixed window of recent neighbors; a few interleaved global layers preserve long-range reach.

The attention mask is banded: token t attends only to the previous W tokens, making per-layer cost O(n x W) instead of O(n^2) and capping that layer's KV cache at W entries (a rolling buffer overwrites older ones). Receptive field still grows with depth — L layers of window W reach ~L x W tokens. Longformer (2020) combined the sliding window with task-specific global tokens; Mistral 7B shipped it in a mainstream LLM (W=4096, rolling-buffer cache). Gemma 2 interleaves local and global layers 1:1 (W=4096); Gemma 3 pushes to 5 local (W=1024) per 1 global layer with only global layers using long-range RoPE, collapsing long-context KV memory (the paper shows cache overhead dropping from ~60% to well under 15% at 32k context). OpenAI's gpt-oss alternates full and W=128 banded layers.

The model learns during pretraining which blocks of context each query needs — near-full-attention quality at a fraction of the FLOPs.

Unlike post-hoc sparsification, the sparsity pattern is part of the architecture and trained end-to-end. NSA (DeepSeek) gives each query three parallel branches — compressed coarse-grained tokens, top-n selected fine-grained blocks, and a sliding window — merged by a learned gate; the blockwise selection is aligned to GPU tensor-core granularity, yielding real wall-clock speedups (up to ~9-11x on 64k-context forward/decode) while matching or beating full attention on benchmarks. MoBA (Moonshot) partitions context into blocks and routes each query to its top-k blocks via affinity between the query and mean-pooled block keys — MoE-style gating applied to attention, hot-swappable with full attention during training. DeepSeek productionized a descendant, DSA (a lightning indexer plus fine-grained top-k token selection), in DeepSeek-V3.2-Exp to cut long-context API costs.

The 2026 refinement of trained sparse attention: stop scoring individual tokens, score compressed blocks of them — and both leading implementations independently landed on a budget of 2,048.

Trained sparse attention (DSA, NSA) put a small learned 'indexer' in front of full attention to pick which past tokens each query may look at. The 2026 versions change the unit of selection from the token to the block. Qwen states the change plainly for its Qwen Sparse Attention: "Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage." Selecting blocks means the indexer scores far fewer candidates, and the tokens it admits arrive contiguously, which is what the memory system actually wants. GLM-5.3-Flash's config reaches the same place from the DSA side: its indexer carries index_kpool 4 with index_kpool_compress true, i.e. keys are pooled into compressed groups of four before scoring, plus index_kpool_always_select_tail true — the most recent block is admitted unconditionally, so local context is never a casualty of a bad routing decision. The indexers themselves are deliberately cheap: Qwen runs its as MQA with 4 query heads over 1 shared key head at head_dim 128; GLM runs 32 indexer heads at head_dim 128 and shares the index across multi-token-prediction iterations (index_share_for_mtp_iteration). The convergence worth noticing is the budget. GLM-5.3-Flash sets index_topk 2048; Qwen3.8-Flash-Next sets indexer_budget 2048 with indexer_compress_ratio 4. Two independent labs, two different codebases, the same number of admitted tokens per query.

2026's other answer to long context: instead of selecting fewer KV entries, shrink what a KV entry is — compress along the sequence axis, or delete the value projection outright.

Sparse attention picks a subset of past tokens to attend to. This family attacks the same cost from the other side: make each cached entry smaller, or make fewer of them exist. Three shipped variants show the range. DeepSeek-V4 interleaves two compressed-KV regimes layer-wise: Compressed Sparse Attention squeezes every m KV tokens into one entry with learned compression weights and then runs top-k lightning-indexer selection over those compressed entries plus a sliding window for local detail, while Heavily Compressed Attention uses the same primitive far more aggressively and then runs ordinary dense attention over the resulting short stream. The step beyond the atlas's existing DSA/NSA entry is that selection now happens over sequence-compressed KV rather than raw KV, and that two differently-compressed regimes coexist in one stack. Compressed Convolutional Attention goes further still: it down-projects queries, keys AND values into a shared latent and performs the entire attention operation inside that latent, never decompressing — which is the sharp difference from MLA, where the latent is a cache format that gets expanded or absorbed for the actual attention. Because Q is compressed too, CCA cuts parameters, cache and attention FLOPs together, so it speeds up prefill and training rather than only decode; it composes with head-sharing as CCGQA. Gemma 4 takes the bluntest route of all and simply deletes the value projection in global layers — 'we improve memory efficiency by re-using keys as values in the global attention layers' — then adds cross-layer KV sharing on the E2B/E4B variants (later layers reuse KV computed earlier, at ratios of 20/35 and 18/42), on top of a 5:1 local-window-to-global interleave and p-RoPE. Google reports these together cut the global KV footprint by up to 37.5%.

Sparse attention makes attention cheap but leaves the indexer that picks the tokens costing O(L²). IndexCache computes it in a minority of layers and lets the rest borrow the answer.

Trained sparse attention replaces the quadratic attention matrix with a top-k selection, but the selector itself still has to score every past token against every query — so the indexer inherits exactly the O(L²) cost the scheme was meant to remove, and at long context it becomes the bottleneck. IndexCache's observation is that the indices chosen at one layer are highly reusable at the next. Layers are split into two roles: 'full' layers run their own lightning indexer, and 'shared' layers skip it entirely and reuse the top-k selection from the nearest full layer above them. The paper reports removing about 75% of indexer compute on a 30B DSA model for 1.82× faster prefill and 1.48× faster decode at negligible quality cost. What makes this atlas-worthy rather than a paper curiosity is that it is already in production at two unrelated vendors — and with the same split. Both Tencent's Hy4-preview and Z.ai's GLM-5.3 carry an indexer_types array of 78 entries reading 21 'full' against 57 'shared'. Two labs, same architecture-level decision, to the layer.

Replace softmax with kernel feature maps or a recurrent state for O(n) attention — now shipped at frontier scale in hybrid layouts.

Linear attention rewrites softmax(QK^T)V as phi(Q)(phi(K)^T V): the d x d running sum phi(K)^T V acts as a recurrent state updated once per token, giving O(n) time and constant memory — no KV cache growth at all. Performer's FAVOR+ constructs phi from random features that unbiasedly approximate the softmax kernel. Pure linear attention loses precise recall, so the 2025-era answer is hybridization: MiniMax-01 interleaves 7 lightning-attention layers (an I/O-aware tiled linear attention) per 1 softmax layer at 456B total parameters; Qwen3-Next-80B mixes Gated DeltaNet linear layers with gated full-attention layers 3:1 (its 'gated attention' — output gating that also eliminates attention sinks — is documented in Qwen's Gated Attention paper); Kimi Linear interleaves Kimi Delta Attention with global MLA 3:1, cutting KV cache ~75%. Notably, MiniMax's later M2 reverted to full attention, so the frontier verdict is still contested.

The founding recipe: a learned gate routes each token to the top-k of N big FFN experts, decoupling parameter count from compute.

A small gating network scores every expert for each token; only the top-k experts (k=2-4 in Shazeer, k=2 in GShard, k=1 in Switch) actually run, and their outputs are combined weighted by the gate scores. Because unchosen experts do no work, total parameters scale with N while per-token FLOPs scale with k. Load is kept even with auxiliary balancing losses and a fixed per-expert capacity factor — tokens overflowing an expert's buffer are dropped and pass through the residual connection. GShard shards experts across accelerators (expert parallelism) so the MoE layer's weights live on different chips and tokens are exchanged via all-to-all.

A small number (8-32) of full-size FFN experts per layer, top-2/top-4 routed, no shared expert — simple, strong, easy to serve.

Every FFN of a decoder-only Transformer is replaced by a MoE layer holding a handful of full-width SwiGLU experts (Mixtral: 8), and a linear router picks the top-2 per token per layer, combining outputs by softmax weight. Mixtral 8x7B activates ~13B of 47B params yet matches much larger dense models; Grok-1 uses the same 8-expert/top-2 shape at 314B. GPT-OSS keeps this shared-expert-free token-choice design but goes wider and finer (gpt-oss-120b: 128 experts, top-4, 5.1B active of 117B), showing the lineage scaled into 2025.

Many small routed experts (activate k of 64-384) plus always-on shared experts, balanced without aux losses — today's dominant open recipe.

DeepSeekMoE segments each FFN into many narrow experts and activates more of them (e.g. 8 of 256 in V3), which multiplies the number of expert combinations and drives specialization; one or two shared experts run on every token to absorb common knowledge so routed experts don't duplicate it. DeepSeek-V3 replaces the auxiliary balancing loss with a per-expert bias added to the routing score and nudged up/down online depending on expert load (aux-loss-free balancing), plus sigmoid affinity scoring and node-limited routing to cap cross-node all-to-all traffic. Kimi K2 (1T total, 32B active, 384 experts, 8 active, 1 shared) and Qwen3-MoE (235B-A22B: 128 fine experts, top-8, but no shared expert and a global-batch balancing loss) are direct descendants; Llama 4 pairs a shared expert with routed experts (Maverick: 128 routed, 17B active / 400B total).

Qwen3.8-Flash-Next carries 51B parameters of n-gram embeddings against 125B of model — a third axis of scale that costs a lookup instead of a matmul, and can live on the host.

MoE decouples parameters from FLOPs by activating a subset of experts. Qwen's argument is that embeddings decouple them further, and better: "Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality." The implementation is a 20,000,000-entry table of bigrams and trigrams (ngram_size 3, ngram_vocab_size_base 20,000,000), injected at a single layer — ple_layer_ids [2], i.e. layer 2 — with heads_per_ngram 8 and the table sharded into split_ngram_parts 128 pieces. The card's own parameter accounting keeps it separate from the model proper: "125B with 6B activated, plus 51B n-gram embedding and 4B MTP". So roughly a third as many parameters as the entire rest of the network sit in a lookup table that never participates in a matrix multiply. Because a table lookup is a gather and not compute, those 51B can sit in host memory and be fetched, which is exactly what "amenable to offloading" is doing in that sentence — the accelerator holds the 125B, the host can hold the rest. The mechanism also rhymes with per-layer embeddings (PLE) as shipped in Gemma 3n, and the config uses ple_* field names for it.

Makes SSM parameters functions of the input so the state can choose what to remember; first attention-free match of Transformers at ~3B.

Mamba makes the SSM parameters Delta, B, C input-dependent, so the recurrence can selectively write, retain, or reset state based on content — recovering the selection ability that time-invariant SSMs like S4 lacked. Input dependence breaks the convolution trick, so Mamba trains with a hardware-aware parallel scan that keeps the expanded state in GPU SRAM, staying O(n) in sequence length. A single homogeneous Mamba block (gated MLP fused with the selective SSM) replaces the attention+MLP pair; there is no KV cache, so generation memory is constant regardless of context length.

Proves selective SSMs and masked linear attention compute the same function, then exploits the duality for 2-8x faster tensor-core training.

Structured State Space Duality shows that an SSM with scalar-times-identity state transitions is exactly a form of masked linear attention (a 1-semiseparable matrix), so one layer has two algorithms: a quadratic attention-like form ideal for short chunks and a linear recurrent form across chunks. Mamba-2 restricts A to a scalar per head, adopts multi-head-style projections, and uses a chunked block decomposition that runs mostly on matmuls — enabling roughly 8x larger recurrent states at similar or better speed than Mamba-1. The duality also unified the SSM and linear-attention research threads, which is why later 'linear attention' hybrids (Gated DeltaNet, Kimi Linear) are siblings of Mamba-2 rather than rivals.

Attention-free RNN trained like a Transformer; evolved from channel-wise decay (v4) to matrix states (v5/6) to delta-rule state edits (v7).

RWKV v4 replaces attention with a 'time-mix' that blends past values under a learned per-channel exponential decay (the WKV operator), keeping training parallelizable while inference runs as a pure RNN with O(1) state per token. v5 'Eagle' upgrades to matrix-valued states (linear-attention-style outer products); v6 'Finch' makes the decay data-dependent via low-rank dynamic projections. v7 'Goose' adopts a generalized delta rule: the state is updated with input-dependent targeted replacement as well as decay, improving in-context state tracking and pushing expressivity beyond what fixed-decay linear layers can do.

DeepMind's RG-LRU gated linear recurrence; Hawk is pure recurrence, Griffin interleaves 2 recurrent blocks per 1 local-attention block.

The Real-Gated Linear Recurrent Unit (RG-LRU) is a diagonal linear recurrence h_t = a_t * h_{t-1} + sqrt(1 - a_t^2) * (i_t * x_t) with a recurrence gate a_t and input gate i_t computed from the input — a gated RNN with no softmax and no state expansion. Hawk stacks RG-LRU residual blocks alone; Griffin interleaves two RG-LRU blocks with one local sliding-window attention block, so the recurrence carries global context while the window provides exact recent-token recall with a bounded KV cache. Inference memory is fixed (state + small window) regardless of context length, and the models matched much more token-hungry Transformer baselines at up to 14B scale.

Mostly-Mamba stacks with a few attention layers kept for exact recall — linear-cost long context that still passes needle tests.

AI21's production hybrid: 1 attention layer per 8, MoE on alternate layers; 256K context with a ~8x smaller KV cache, fitting a single 80GB GPU at up to ~140K-token inputs. Zyphra's Mamba tower with ONE shared attention block re-applied every ~6 layers — repeated global attention for the parameter cost of one block. Microsoft's strictly-linear hybrid: Mamba -> MLP -> sliding-window attention -> MLP per unit; trained at 4K, streams past 1M tokens. Mamba-2-heavy hybrids (~92% of layers) at 8B-56B: ~3x faster inference at similar accuracy; Nano 2 tunes the recipe for reasoning. 9 Mamba-2 blocks per 1 attention block, no positional encodings, MoE variants; IBM reports >70% serving-memory reduction. Apache 2.0. Falcon-H1 instead runs attention and Mamba-2 heads in parallel inside every block.

2025's dominant hybrid recipe: 3 gated-DeltaNet linear layers per 1 attention layer; delta-rule state edits plus Mamba-2-style decay.

Gated DeltaNet (NVIDIA/MIT, 2024) combines the delta rule — a rank-1 targeted edit that overwrites the old value stored under a key in the fixed-size matrix state, S_t = a_t * S_{t-1}(I - b_t k_t k_t^T) + b_t v_t k_t^T — with Mamba-2-style global decay gating, beating both Mamba-2 and plain DeltaNet. Qwen3-Next 80B-A3B interleaves three Gated DeltaNet layers per one gated full-attention layer with an ultra-sparse MoE (3B active), reporting roughly 10x throughput versus Qwen3-32B beyond 32K context; Kimi Linear (Moonshot, 2025) refines the operator into Kimi Delta Attention (finer channel-wise decay, chunked kernels) at a 3:1 KDA:MLA ratio in a 48B-A3B model, cutting KV cache by up to 75%. Via the SSD duality these are siblings of Mamba-2 — the SSM and linear-attention branches have effectively merged.

Diffusion's second act in language: not as the model, but as the draft model. A tiny block-diffusion network proposes a whole block of tokens in one pass for an ordinary autoregressive model to verify.

Speculative decoding needs a cheap drafter whose guesses a big model can verify in parallel; the usual drafter is a small autoregressive model, which still emits one token per forward pass. DFlash makes the drafter a block diffusion model instead: it predicts an entire block of positions simultaneously from a masked input, keeps the top candidates at each position, and a selector traces a coherent path through them for the target model to check. Because verification is lossless — the target model accepts only what it would have produced anyway — output distribution is unchanged. DFlash 2 adds two-tap dynamic convolutions to counter quality decay toward the end of a block. This matters for a field guide because it inverts the usual relationship between the two paradigms: the atlas's existing diffusion-language-models entry covers diffusion as the language model, an approach that has struggled to displace autoregression on quality. Here diffusion does the one thing it is unambiguously better at — filling many positions at once — while autoregression keeps final say. The drafters are real artifacts you can download: Meta ships Muse-Glimmer-30B-assistant (2.56B, block_size 16, 5 layers, reading the target's hidden states at layers 1/13/25/37/49), and Inco AI publishes drafters fitted to other vendors' models, including Qwen3.8-27B and GLM-5.3-Flash.

A small extra head trained to predict token t+2 onward, so the model drafts its own continuation and verifies it in one pass. By 2026 it is near-universal — and the newest variant makes one head recursive instead of training one per offset.

Speculative decoding needs a drafter. MTP makes the model its own: during training, extra heads predict tokens beyond the immediate next one, and at inference those predictions become a draft the main model verifies in parallel. It is now standard equipment — the field guide's own config work found `num_nextn_predict_layers` in GLM-5.3-Flash, Ling-3.0-flash, Nemotron-3.5-Lightning, Hy4 and Qwen3.8-Flash-Next, and its absence in the released Kimi K3 checkpoint was notable enough to record. The 2026 refinement addresses a train/inference mismatch that the original DeepSeek-V3-style design has: each offset head is trained under ground-truth hidden states, but at inference it must condition on its own generated ones, and the mismatch compounds with draft length. Nemotron 3 Super's answer is to stop training a separate module per offset. It trains two MTP layers that share one parameter set and applies the same head recursively at inference, feeding it its own generated hidden state to extend the draft. NVIDIA's stated rationale is that the shared-weight formulation regularizes the head across prediction horizons and makes it robust to exactly the self-generated states it will meet, and the reported result is an average acceptance length of 3.45 at draft length 7, with degradation at draft indices 4-7 'substantially milder than with independently trained offset heads'. The config shows the shape: `num_nextn_predict_layers: 1` with `mtp_hybrid_override_pattern: "*E"` — the MTP module is itself one attention plus one MoE layer — against 88 hidden layers.

Moonshot AI's 2.78T-parameter sparse-MoE flagship — open weights since 2026-07-27, natively multimodal, 1M context. Every headline number below is now config-verified.

Kimi K3 shipped its open weights on 2026-07-27, so the architecture is no longer announcement-level: it can be read straight from config.json. The released checkpoint is a 93-layer, 2,779,931,837,184-parameter (2.78T) model — the safetensors metadata shows ~2.72T of those params stored as U8, i.e. it ships natively 8-bit quantized (the card is tagged compressed-tensors / 8-bit). It is not text-only: the architecture class is KimiK3ForConditionalGeneration with a vision tower (patchmergerv2 projector, mm_hidden_size 1024, 2×2 patch merging) whose position embeddings carry a time axis — so image and video in, text out (pipeline tag image-text-to-text). The MoE is exactly what was reported and is now confirmed: num_experts 896 with num_experts_per_token 16, plus num_shared_experts 2 (a figure Moonshot never published), sigmoid router scoring with aux-loss-free noaux_tc top-k selection and latent_moe_use_norm — the "Stable LatentMoE" branding. Attention is the KDA hybrid, and the config settles the ratio arithmetically: linear_attn_config lists 69 KDA layers against 24 full-attention layers placed at every 4th position (4, 8, 12 … 92), i.e. 2.88:1 — the 3:1 pattern Moonshot showed in its blog diagram. KDA runs 96 heads at head_dim 128 with a 4-wide short convolution and a full-rank gate; the full-attention layers are MLA (kv_lora_rank 512, q_lora_rank 1536, 96 heads, 128 NoPE + 64 RoPE dims). Attention Residuals are real and parameterized: attn_res_block_size 12. Context is 1,048,576 tokens, vocab 163,840. Note the released config sets num_nextn_predict_layers 0 — no multi-token-prediction head in this checkpoint.

Z.ai's 753B-A40B MIT MoE flagship for long-horizon coding/agents: DSA sparse attention + IndexShare indexer reuse on a solid 1M-token context.

GLM-5.2 is a 78-layer `glm_moe_dsa` transformer: each layer pairs MLA (q_lora_rank 2048 / kv_lora_rank 512 latents, 64 heads, 256-dim QK = 192 NoPE + 64 decoupled RoPE) with a DeepSeek-Sparse-Attention core where a lightweight "lightning indexer" (32 heads x 128 dim) scores past tokens and core attention runs only over the top-2048 (index_topk), cutting attention from O(L^2) to O(Lk). The DSA indexer itself stays O(L^2) and in GLM-5/5.1 ran at every layer, so IndexShare (the linked paper is titled "IndexCache", arXiv:2603.12201) runs real indexers only in "full" layers — layers 0-2 plus every 4th (index_topk_freq=4; 21 of 78) — while the 57 "shared" layers reuse the nearest full layer's top-k indices; in the paper's training-aware variant, retained indexers are distilled against the averaged attention distributions of the layers they serve (the likely deployed recipe, though Z.ai doesn't say outright — see undisclosed). Z.ai claims 2.9x per-token FLOPs reduction at 1M context (vendor-only figure, HF card + NIM); the paper independently measured up to 1.82x prefill / 1.48x decode and 75% indexer-compute removal on a 30B DSA model. The FFN side is DeepSeek-V3-style MoE: 3 dense layers then 75 MoE layers with 256 routed experts (8 active) + 1 shared expert, sigmoid scoring with aux-loss-free `noaux_tc` routing — 753B total all-in; the GLM-5 report's 744B figure covers the main model incl. embeddings/output head but excl. the ~9.9B MTP layer (convention derived from config-param math — the report doesn't state it), 40B active (active figure stated for GLM-5, whose config shape is identical). One MTP layer (num_nextn_predict_layers=1; the GLM-5 report trains 3 MTP layers sharing one parameter set) drives speculative decoding and also reuses the shared indices (`index_share_for_mtp_iteration: true`); Z.ai claims up to 20% longer acceptance length vs 5.1. Context is 1,048,576 positions (rope_theta 8M, up from 202,752 / theta 1M in GLM-5/5.1) with 128K max output on the API; "thinking effort" is purely an inference-time API control (`reasoning_effort`), not an architecture mechanism.

The rare frontier release whose card prints its own block layout. Twelve repeats of three Gated-DeltaNet blocks to one sparse-attention block, 512 experts, and a fourth parameter budget in an n-gram table.

Qwen publishes the layout as a formula — "12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))" — and the config agrees exactly: 48 layers with full_attention_interval 4 and a layer_types array that runs linear_attention three times, then full_attention, twelve times over. That is the same 3:1 ratio Kimi K3 uses for KDA:MLA (69:24) and that Qwen3-Next and Kimi Linear established, now stated by the vendor rather than inferred. The linear path is Gated DeltaNet with a deliberately asymmetric head split — 48 value heads against 16 query/key heads, head dim 128, a 4-wide short convolution and a sigmoid output gate. The full-attention path is Qwen Sparse Attention, whose indexer is MQA with 4 query heads over 1 shared key head at head_dim 128 and a 2,048-token budget. Every one of the 48 blocks is followed by a MoE layer: 512 experts, 10 routed plus 1 shared, expert intermediate dimension 640 — very fine-grained, the DeepSeek-style direction taken further. Two parameter pools sit outside that: a 20M-entry n-gram embedding table worth 51B parameters injected at layer 2, and a 4B single-layer MTP head trained with multiple steps. Total accounting from the card: 125B parameters, 6B activated, plus 51B n-gram, plus 4B MTP. The checkpoint is natively multimodal (a vision tower and interleaved mRoPE with mrope_section [11, 11, 10]) and its config sets max_position_embeddings 262,144, while Qwen states the productized Qwen3.8-Flash build defaults to 1M context.

Bolt a pretrained ViT onto an LLM with a small trained projector; image features enter the prompt as soft tokens.

A contrastively pretrained ViT (CLIP or SigLIP) encodes the image into patch features; a small connector — a linear layer or 2-layer MLP in LLaVA, a single-layer cross-attention resampler compressing to 256 tokens in Qwen-VL — maps them into the LLM's embedding space, where they are spliced into the token sequence like ordinary word embeddings. Training is staged: first align only the projector on image-caption pairs, then visual instruction-tune with the LLM (and often later the ViT) unfrozen. Modern variants handle native/dynamic resolution via tiling (InternVL 1.5) or M-RoPE with a ViT trained alongside the LLM (Qwen2-VL), and compress patches (2x2 token merge, pixel-shuffle) to keep image token counts manageable.

Vision enters through gated cross-attention layers inserted between frozen LM blocks — Flamingo's recipe, revived by Llama 3.2 Vision.

A frozen vision encoder feeds a Perceiver Resampler that compresses any number of images or video frames into a fixed set of 64 latent tokens (Flamingo). Newly inserted gated cross-attention-dense layers, interleaved between the frozen LM's blocks, read those latents as keys/values; a tanh gate initialized at zero means the model starts out exactly equal to the original text LM and learns to open the visual pathway during training. Llama 3.2 Vision (disclosed in The Llama 3 Herd of Models) applies the same compositional idea: a ViT encoder plus cross-attention layers inserted every 4th decoder layer, with the text weights kept frozen so text-only performance is provably unchanged. Image information never occupies positions in the token sequence.

One transformer, one stream: images (and audio) become tokens or patches in the same sequence from pretraining day one.

Instead of grafting vision onto a text model, one transformer is trained from the start on interleaved multimodal sequences. Chameleon VQ-quantizes each 512x512 image into 1024 discrete tokens from an 8192-entry codebook and trains a single autoregressive model over mixed text+image tokens — requiring QK-norm and revised norm placement for stability at scale — so the same softmax can emit words or pixels. Fuyu goes further and deletes the encoder entirely: raw image patches are linearly projected straight into the decoder's first layer at arbitrary resolution, with an image-newline token marking row breaks. GPT-4o and Gemini are disclosed only at the capability level — a single model trained end-to-end (OpenAI) / jointly from the start (Google) across text, vision, and audio — with internal mechanics undisclosed.

Omni models take speech, audio, image, video, and text in — and stream speech out — replacing the ASR → LLM → TTS pipeline.

GPT-4o's system card discloses a single autoregressive model that accepts any mix of text, audio, image, and video and emits text, audio, and image from the same network, collapsing voice latency to roughly 300ms versus the old three-model ASR→LLM→TTS pipeline; internals are undisclosed. Qwen2.5-Omni discloses a full open recipe: a Thinker (LLM decoder) ingests all modalities — audio and video time-aligned by TMRoPE, a time-interleaved multimodal RoPE — and produces text plus high-level hidden states, while a Talker, a dual-track autoregressive speech-token model, streams speech tokens conditioned on those states through a sliding-window DiT vocoder for real-time audio. Moshi runs full-duplex: one backbone (Helium 7B) jointly models the user's and the system's audio streams via the Mimi neural codec plus an inner-monologue text stream, reaching ~160ms theoretical latency.

Text generated by iteratively denoising/unmasking whole blocks in parallel, instead of predicting one token at a time.

Instead of left-to-right next-token prediction, the model is trained on a forward corruption process (randomly masking or noising tokens) and learns the reverse process: given a partially masked sequence, predict all masked tokens simultaneously. Generation starts from a fully masked block and runs a few refinement steps, each one committing the most confident tokens and re-predicting the rest. The backbone is still a standard Transformer, but with bidirectional (non-causal) attention, and speed comes from emitting many tokens per forward pass. LLaDA showed this trained-from-scratch at 8B rivals autoregressive peers; Mercury and Gemini Diffusion showed commercial-grade throughput (1,000+ tokens/sec).

Transformers trained from scratch with ternary {-1,0,+1} weights, so matmuls become adds — ~10x less memory and energy.

BitNet replaces every nn.Linear in the Transformer with a BitLinear layer whose weights are constrained to {-1, 0, +1} (~1.58 bits) during training itself — quantization is native, not applied after the fact, so the model learns around the constraint (a straight-through estimator passes gradients while latent full-precision weights are kept for the optimizer). Activations are quantized to 8 bits. Because weights are ternary, matrix multiplication reduces to integer additions and sign flips, slashing memory, energy, and latency. The 2024 paper showed ternary matches FP16 quality from ~3B params; Microsoft's open BitNet b1.58 2B4T (2B params, 4T tokens) ships with a CPU-friendly runtime at ~0.4 GB memory.

o1/R1-class models are the same transformer architecture; the novelty is RL training that elicits long private chain-of-thought.

Honest note for the field guide: despite the branding, o1/R1-style 'reasoning models' are not a new neural architecture. The DeepSeek-R1 paper is explicit that R1 is DeepSeek-V3-Base — a standard MoE transformer (671B total / 37B active) — post-trained with reinforcement learning (GRPO) using rule-based rewards, which spontaneously incentivizes long chain-of-thought, self-verification, and backtracking (R1-Zero used pure RL, R1 added a small cold-start SFT stage). OpenAI's o1 System Card likewise describes large-scale RL to 'think before answering' via chain-of-thought, disclosing nothing about a changed architecture. What changes at inference is behavior and compute allocation (many more generated thinking tokens), not the network.

The exact parameter count of GPT-6 Astra and Claude Fable 5.1 is written down, in a legally-mandated form, sitting with a regulator — and there is no public route to it. 'Undisclosed' in 2026 means withheld by design, not unrecorded.

A field guide has to say something honest about models whose weights nobody outside the lab has seen, and the honest thing changed in 2026. It is no longer true that nobody wrote the numbers down. Under the EU AI Act, providers of general-purpose AI models must prepare Annex XI technical documentation, and the GPAI Code of Practice's Model Documentation Form has a field for the total parameter count 'to at least two significant figures' alongside training compute in FLOPs, training time and energy. Both OpenAI and Anthropic are signatories. So the numbers exist, on a form, today. The catch is routing: that form goes to the AI Office and national competent authorities, and is released only on their request. The single artifact that must be public — the Article 53(1)(d) training-content summary — was designed without an architecture field at all. OpenAI's public summary duly gives real bucketed data volumes (text >10 trillion tokens, images >1 billion, audio and video >1 million hours each) and the useful statement that Astra 'is not a modification or a fine-tune of a prior model', and says nothing whatever about the model's shape. Anthropic states its reasoning outright in the Fable 5.1 system card: it declines to disclose 'competitively sensitive information'. The other channels are exhausted too, and their emptiness is itself informative. Patents: OpenAI's estate covers a modality-routed MoE, chain-of-thought serving with a second summarizing model, and prefix KV-cache reuse — but a patent claims an invention, never a shipped product, and Anthropic's ~20 filings turn out to be an inherited Adept portfolio describing Fuyu-8B, not Claude. Interpretability papers, historically a leak, have been closed off: Anthropic now reports layer positions as percentages of '25 evenly spaced layers … reindexed to the range [0-100]'. What remains is mechanism-level disclosure — serving topology, tokenizer changes, decode-time watermarking, effort semantics, probes on live activations — and that is what this atlas records for closed models instead of a blank cell.

If the vendor will not say how wide or deep the model is, measure it. A 2026 method recovers hidden dimension, depth and parameter count through the restrictive APIs that were meant to prevent exactly that.

Carlini et al. showed in 2024 that a model's hidden dimension leaks through its logit API: the outputs of a final linear layer lie on a low-rank subspace, so enough queries reveal the rank — they recovered 1024 for ada and babbage. Vendors responded by closing the surface, removing logit-bias and full-logprob access, which shut those attacks down. NightVision (2026) attacks the restricted regime that remains: it uses common-set prompting plus spectral analysis to recover hidden dimension, and time-to-first-token scaling to estimate depth and parameter count, under single-logit output and no logit bias. This belongs in a field guide for two opposite reasons. First, it is the only route by which anyone outside a lab could ever put a number on GPT-6 Astra or Claude Fable 5.1 — the disclosure regime having closed every other one. Second, it is a discipline check: the method reports roughly 23% mean relative error, so it produces an order of magnitude and an error bar, not a spec sheet. A measurement with stated error is a different object from a leaked figure with none, and only the first kind is worth recording — with its error bar attached.

Sub-3B models engineered for phones: at small scale, deep-and-thin beats wide-shallow, and data curation does the heavy lifting.

MobileLLM (Meta) systematically studied sub-billion-parameter design and found the opposite of conventional wisdom at that scale: allocating parameters to depth (more, thinner layers) beats width, and combining that with embedding sharing, grouped-query attention, and immediate block-wise layer sharing (reusing a block twice before moving on) yields 2-4+ point accuracy jumps over prior 125M/350M models with no extra memory. The SmolLM line (Hugging Face) attacks the same target from the data side: SmolLM2-1.7B is a fairly standard architecture trained on ~11T tokens with multi-stage rebalancing of curated datasets (FineMath, Stack-Edu, SmolTalk), showing meticulous data-centric training makes small models punch far above their weight. SmolLM3-3B adds GQA plus NoPE-style positional handling and dual-mode (think/no-think) reasoning at 3B.

One shared stack of layers applied recursively, with a router giving each token its own number of passes through it.

MoR unifies two efficiency ideas in one recursive Transformer: parameter sharing and adaptive compute. A single block of layers is reused up to N times (recursion), so the parameter count stays small; a lightweight router then decides per token how many recursion steps that token gets — easy tokens exit after one pass, hard tokens loop more, in the spirit of mixture-of-experts routing but over depth. Recursion-wise KV caching keeps attention cost consistent with each token's actual depth. At 135M-1.7B scale it forms a new Pareto frontier over vanilla and prior recursive baselines at equal training FLOPs, with higher inference throughput.

Run a block of layers around a loop instead of stacking more of them: depth becomes a runtime dial rather than a weight budget. Real, shipped at small scale — and the mechanism a widely-circulated leak alleges for GPT-6 Astra, on evidence that does not stand up.

A looped transformer applies one weight-tied block repeatedly, so effective depth is decoupled from parameter count and can be varied at inference. Ouro-2.6B runs 48 layers looped 4x with deep supervision; Huginn-0125 recurs a 4-layer core 16 times. Retrofitting shows the idea does not require training from scratch: a pretrained Qwen2.5-0.5B can be split into Prelude / weight-tied Recurrent Block / Coda with an identity-preserving one-loop path, installed with as few as 6M trained parameters over frozen base weights, and it then extrapolates to about 1.5x the supervised depth — beating a same-size scratchpad model 84% to 72% overall, and 53% to 2.5% beyond depth 10, while running 7.6x faster. Note the weakness the paper itself carries: one base model, at 0.5B. The interesting 2026 result is what looping does to interpretability, which matters because chain-of-thought monitoring is now a safety-critical dependency for every frontier lab. Testing Ouro and Huginn against a conventional 64-layer baseline, a global workspace does still form in the iterated part — but recurrence changes how it can be accessed: Ouro reconstructs workspace content on every loop, so linear transport cannot cross loop boundaries and any intervention must span every remaining loop, while Huginn carries content across all sixteen recurrences but reads and writes act only within a sliding window of about two. In other words, if a model loops, what a monitor can see and steer changes shape. That is why the persistent claim that GPT-6 Astra uses recurrent depth matters — and why this entry does not assert it. It originates in a single anonymously-sourced press report behind a paywall that no one in this project's research could read; there is no technical report, no code, no patent and no vendor confirmation, and the one quantitative statement OpenAI's Chief Scientist has made publicly — that the computation graph depth of present frontier models is within a factor of two of GPT-4 — sits awkwardly with heavy looping rather than supporting it.

Tokenizer-free LLM over raw bytes: entropy-based dynamic patches feed a big latent transformer; matches Llama 3 with fewer FLOPs.

BLT (Meta) removes the tokenizer entirely and operates on raw bytes, solving the compute problem with a three-part design: a small local encoder groups bytes into dynamically sized patches — patch boundaries are placed where a tiny byte-LM's next-byte entropy is high, so predictable stretches get long patches and hard spots get short ones — a large latent transformer runs only over patch representations, and a small local decoder maps latent outputs back to bytes. Because patches average longer than BPE tokens where text is easy, compute is allocated to where prediction is genuinely hard. Scaled to 8B parameters and 4T training bytes, BLT matches tokenization-based Llama-3-style training while cutting inference FLOPs up to ~50%, and is markedly more robust to spelling noise, character-level tasks, and rare scripts.

TypeSafe’s Jev answers a typed question with a probability distribution over options you supplied, in one parallel pass. Nothing about its internals is published — but the output contract alone changes what the model can be used for. TypeSafe positions it as the first "System One model", a category it is also naming.

Every architecture elsewhere in this atlas is a way of producing the next token faster or cheaper. This one is defined by refusing to produce tokens at all. A request carries one `state` — the thing being judged — plus a map of named questions, each of one of three types: a Noul (yes/no, returning the probability of yes), a Choice (one option from a set of up to 255, returning the full distribution), or a Score (a position along 2–10 ordered levels you describe, returned as the probability-weighted mean of the level indices). The state is ingested once and every question is evaluated against it in the same parallel pass, independently: one question’s answer is never context for another, so adding a question changes neither the other answers nor, materially, the latency. The answer is constrained to the option set supplied, which makes a type error structurally impossible and removes the parsing step entirely. What is NOT disclosed is almost everything a reader of this atlas would want: no parameter count, no layer count, no attention mechanism, no training compute. TypeSafe describes the sampling as "parallel" rather than sequential and says outputs are "type-safe structured values" whose "possible outputs and structure are defined in advance", and it names its training method RLCD — Reinforcement Learning for Calibrated Decisions — positioned against RLHF (optimizing human preference) and RLVR (optimizing verifiable correctness), aimed instead at "epistemically honest probabilities". Beyond those characterizations there is nothing to verify, and "non-autoregressive" should be read as the vendor’s own description rather than an established structural fact. The honest reading is that this is an output-contract innovation whose implementation is closed, launched 2026-09-15 and gated behind a waitlist that itself limits outside scrutiny. The calibration claim is the load-bearing one and the least tested. Every Choice and Score answer carries a `confidence` in [0,1] summarizing how peaked the distribution is; TypeSafe does not publish the formula, though the interactive widget in its own docs gives the three-option case as (3·p_max − 1)/2, i.e. normalized peak probability, which reproduces 12 of the 14 worked examples in the documentation exactly and misses two Score examples. Nouls deliberately carry no confidence at all, because a two-outcome distribution is fully described by its single value. No calibration curve has been published by TypeSafe or by anyone else, so whether these probabilities track real-world frequencies is untested in public — which matters, because calibration is the entire product.