Model architectures

The 2017 seq2seq Transformer: a bidirectional encoder reads the source, a causal decoder writes the target via cross-attention.

The encoder stack applies full bidirectional self-attention over the input sequence to produce contextual representations in one parallel pass. The decoder stack autoregressively generates the output: each decoder layer runs causal (masked) self-attention over tokens generated so far, then cross-attention where decoder queries attend over the encoder's output keys/values, then a feed-forward block. Sinusoidal positional encodings inject order; multi-head attention lets different heads specialize. This replaced recurrence entirely, making training parallelizable across sequence positions and unlocking the scaling that all later LLMs build on.

Bidirectional encoder pretrained by masking tokens; still the workhorse for embeddings, rerankers, and classifiers, revived by ModernBERT.

Take only the Transformer's encoder stack, so every token attends to every other token in both directions, and pretrain with masked language modeling: randomly mask ~15% of tokens and predict them from full bidirectional context. The result is a deep contextual representation of the whole input in a single parallel forward pass — no autoregressive loop. Downstream you attach a small head (a classifier on [CLS], a token tagger, or a pooling layer for embeddings) and fine-tune. RoBERTa showed BERT was undertrained (more data/steps, dynamic masking, drop NSP); DeBERTa added disentangled content/position attention; ModernBERT (2024) retrofitted the recipe with RoPE, GeGLU, alternating local/global attention, Flash Attention, 8k context, and 2T training tokens.

T5/BART framed every NLP task as text-in, text-out with a seq2seq Transformer; Flan-T5, UL2, and 2025's T5Gemma keep the line alive.

Keep both Transformer stacks but pretrain on denoising: T5 masks contiguous spans and trains the decoder to emit the missing spans (span corruption), casting every task — translation, summarization, QA, classification — as 'text in, text out' with a task prefix. BART instead corrupts whole documents (deletion, shuffling, infilling) and reconstructs them, which transfers especially well to summarization. UL2 unified the objectives as mixture-of-denoisers (short spans, long spans, prefix-LM) switchable by a mode token. The 2025 revival, T5Gemma (paper title 'Encoder-Decoder Gemma'), adapts pretrained decoder-only Gemma 2 checkpoints into encoder-decoder form via UL2/prefix-LM continued pretraining, showing better quality-per-inference-FLOP than the source decoder-only models on input-heavy tasks.

The 2018-2020 OpenAI recipe: causal decoder stack with learned absolute positions, LayerNorm, GELU MLPs, standard multi-head attention.

Token IDs are embedded via a BPE vocabulary and summed with a learned absolute position embedding table, then pushed through N identical blocks: causally-masked multi-head self-attention followed by a 4x-width two-layer GELU MLP, each wrapped in a residual connection with LayerNorm. GPT-1 placed LayerNorm after each sublayer (post-norm, as in the 2017 Transformer decoder minus cross-attention); GPT-2 moved it before each sublayer and added a final LayerNorm, which is what made 48+ layer training stable. A final LayerNorm feeds an unembedding projection (weight-tied to the input embedding) producing next-token logits; the entire training objective is next-token prediction. GPT-3 kept this recipe at 175B, alternating dense and locally-banded sparse attention layers.

The post-2023 dense standard: pre-RMSNorm, rotary positions (RoPE), SwiGLU gated MLPs, grouped-query attention, no bias terms.

Keeps the GPT causal-decoder skeleton but swaps every component: RMSNorm (rescale by root-mean-square only, no mean-centering or bias) is applied before each sublayer; positions are injected inside attention by rotating query/key vectors with position-dependent angles (RoPE), giving relative encoding and enabling long-context extension via theta rescaling. The MLP becomes SwiGLU — SiLU(gate_proj(x)) * up_proj(x) fed to down_proj, with hidden width ~8/3·d to hold parameter count. Attention uses grouped-query attention: many query heads share a small set of K/V heads, shrinking the KV cache several-fold with near-MHA quality; linear layers drop bias terms. Llama 1 established pre-RMSNorm+RoPE+SwiGLU (still MHA); Llama 2 70B and Mistral 7B added GQA; Llama 3 applied GQA at every size with a 128K vocab and RoPE theta 500k.

Share key/value heads across query heads — same attention math, 8-64x smaller KV cache for fast batched decoding.

In multi-head attention (MHA) every one of the h query heads has its own K and V head, so the decode-time KV cache costs 2 x n_layers x n_kv_heads x d_head x bytes per token — for a Llama-2-70B-shaped model (80 layers, 64 heads, d_head 128, fp16) that is ~2.6 MB per token, which dominates GPU memory and bandwidth at long context and large batch. MQA (Shazeer 2019) keeps h query heads but a single shared K/V head, cutting the cache by h-fold (~41 KB/token in the same config) at some quality cost. GQA interpolates: query heads are partitioned into g groups, each sharing one KV head (Llama 2 70B uses g=8, ~0.33 MB/token), recovering near-MHA quality; the GQA paper also showed an existing MHA checkpoint can be converted by mean-pooling its KV heads and uptraining with ~5% of original compute.

Compress each token's K/V into one small latent vector, reconstruct full heads on the fly — ~93% KV-cache cut without losing head diversity.

Instead of caching per-head keys and values, a learned down-projection compresses the token's hidden state into a single latent c_KV (DeepSeek-V2: 512 dims) plus a small decoupled RoPE key (64 dims) — only ~576 floats per token per layer are cached, versus 2 x 128 heads x 128 dims = 32,768 for equivalent MHA (~57x smaller per layer; DeepSeek reports 93.3% total KV reduction, comparable to GQA with just 2.25 groups). At attention time, up-projection matrices reconstruct all 128 per-head K/V — and because these are linear, they can be absorbed into the query and output projections so inference attends over the latent directly. The RoPE path must be decoupled because rotary position rotation does not commute with the low-rank compression.

Each token attends only to a fixed window of recent neighbors; a few interleaved global layers preserve long-range reach.

The attention mask is banded: token t attends only to the previous W tokens, making per-layer cost O(n x W) instead of O(n^2) and capping that layer's KV cache at W entries (a rolling buffer overwrites older ones). Receptive field still grows with depth — L layers of window W reach ~L x W tokens. Longformer (2020) combined the sliding window with task-specific global tokens; Mistral 7B shipped it in a mainstream LLM (W=4096, rolling-buffer cache). Gemma 2 interleaves local and global layers 1:1 (W=4096); Gemma 3 pushes to 5 local (W=1024) per 1 global layer with only global layers using long-range RoPE, collapsing long-context KV memory (the paper shows cache overhead dropping from ~60% to well under 15% at 32k context). OpenAI's gpt-oss alternates full and W=128 banded layers.

The model learns during pretraining which blocks of context each query needs — near-full-attention quality at a fraction of the FLOPs.

Unlike post-hoc sparsification, the sparsity pattern is part of the architecture and trained end-to-end. NSA (DeepSeek) gives each query three parallel branches — compressed coarse-grained tokens, top-n selected fine-grained blocks, and a sliding window — merged by a learned gate; the blockwise selection is aligned to GPU tensor-core granularity, yielding real wall-clock speedups (up to ~9-11x on 64k-context forward/decode) while matching or beating full attention on benchmarks. MoBA (Moonshot) partitions context into blocks and routes each query to its top-k blocks via affinity between the query and mean-pooled block keys — MoE-style gating applied to attention, hot-swappable with full attention during training. DeepSeek productionized a descendant, DSA (a lightning indexer plus fine-grained top-k token selection), in DeepSeek-V3.2-Exp to cut long-context API costs.

Replace softmax with kernel feature maps or a recurrent state for O(n) attention — now shipped at frontier scale in hybrid layouts.

Linear attention rewrites softmax(QK^T)V as phi(Q)(phi(K)^T V): the d x d running sum phi(K)^T V acts as a recurrent state updated once per token, giving O(n) time and constant memory — no KV cache growth at all. Performer's FAVOR+ constructs phi from random features that unbiasedly approximate the softmax kernel. Pure linear attention loses precise recall, so the 2025-era answer is hybridization: MiniMax-01 interleaves 7 lightning-attention layers (an I/O-aware tiled linear attention) per 1 softmax layer at 456B total parameters; Qwen3-Next-80B mixes Gated DeltaNet linear layers with gated full-attention layers 3:1 (its 'gated attention' — output gating that also eliminates attention sinks — is documented in Qwen's Gated Attention paper); Kimi Linear interleaves Kimi Delta Attention with global MLA 3:1, cutting KV cache ~75%. Notably, MiniMax's later M2 reverted to full attention, so the frontier verdict is still contested.

The founding recipe: a learned gate routes each token to the top-k of N big FFN experts, decoupling parameter count from compute.

A small gating network scores every expert for each token; only the top-k experts (k=2-4 in Shazeer, k=2 in GShard, k=1 in Switch) actually run, and their outputs are combined weighted by the gate scores. Because unchosen experts do no work, total parameters scale with N while per-token FLOPs scale with k. Load is kept even with auxiliary balancing losses and a fixed per-expert capacity factor — tokens overflowing an expert's buffer are dropped and pass through the residual connection. GShard shards experts across accelerators (expert parallelism) so the MoE layer's weights live on different chips and tokens are exchanged via all-to-all.

A small number (8-32) of full-size FFN experts per layer, top-2/top-4 routed, no shared expert — simple, strong, easy to serve.

Every FFN of a decoder-only Transformer is replaced by a MoE layer holding a handful of full-width SwiGLU experts (Mixtral: 8), and a linear router picks the top-2 per token per layer, combining outputs by softmax weight. Mixtral 8x7B activates ~13B of 47B params yet matches much larger dense models; Grok-1 uses the same 8-expert/top-2 shape at 314B. GPT-OSS keeps this shared-expert-free token-choice design but goes wider and finer (gpt-oss-120b: 128 experts, top-4, 5.1B active of 117B), showing the lineage scaled into 2025.

Many small routed experts (activate k of 64-384) plus always-on shared experts, balanced without aux losses — today's dominant open recipe.

DeepSeekMoE segments each FFN into many narrow experts and activates more of them (e.g. 8 of 256 in V3), which multiplies the number of expert combinations and drives specialization; one or two shared experts run on every token to absorb common knowledge so routed experts don't duplicate it. DeepSeek-V3 replaces the auxiliary balancing loss with a per-expert bias added to the routing score and nudged up/down online depending on expert load (aux-loss-free balancing), plus sigmoid affinity scoring and node-limited routing to cap cross-node all-to-all traffic. Kimi K2 (1T total, 32B active, 384 experts, 8 active, 1 shared) and Qwen3-MoE (235B-A22B: 128 fine experts, top-8, but no shared expert and a global-batch balancing loss) are direct descendants; Llama 4 pairs a shared expert with routed experts (Maverick: 128 routed, 17B active / 400B total).

Makes SSM parameters functions of the input so the state can choose what to remember; first attention-free match of Transformers at ~3B.

Mamba makes the SSM parameters Delta, B, C input-dependent, so the recurrence can selectively write, retain, or reset state based on content — recovering the selection ability that time-invariant SSMs like S4 lacked. Input dependence breaks the convolution trick, so Mamba trains with a hardware-aware parallel scan that keeps the expanded state in GPU SRAM, staying O(n) in sequence length. A single homogeneous Mamba block (gated MLP fused with the selective SSM) replaces the attention+MLP pair; there is no KV cache, so generation memory is constant regardless of context length.

Proves selective SSMs and masked linear attention compute the same function, then exploits the duality for 2-8x faster tensor-core training.

Structured State Space Duality shows that an SSM with scalar-times-identity state transitions is exactly a form of masked linear attention (a 1-semiseparable matrix), so one layer has two algorithms: a quadratic attention-like form ideal for short chunks and a linear recurrent form across chunks. Mamba-2 restricts A to a scalar per head, adopts multi-head-style projections, and uses a chunked block decomposition that runs mostly on matmuls — enabling roughly 8x larger recurrent states at similar or better speed than Mamba-1. The duality also unified the SSM and linear-attention research threads, which is why later 'linear attention' hybrids (Gated DeltaNet, Kimi Linear) are siblings of Mamba-2 rather than rivals.

Attention-free RNN trained like a Transformer; evolved from channel-wise decay (v4) to matrix states (v5/6) to delta-rule state edits (v7).

RWKV v4 replaces attention with a 'time-mix' that blends past values under a learned per-channel exponential decay (the WKV operator), keeping training parallelizable while inference runs as a pure RNN with O(1) state per token. v5 'Eagle' upgrades to matrix-valued states (linear-attention-style outer products); v6 'Finch' makes the decay data-dependent via low-rank dynamic projections. v7 'Goose' adopts a generalized delta rule: the state is updated with input-dependent targeted replacement as well as decay, improving in-context state tracking and pushing expressivity beyond what fixed-decay linear layers can do.

DeepMind's RG-LRU gated linear recurrence; Hawk is pure recurrence, Griffin interleaves 2 recurrent blocks per 1 local-attention block.

The Real-Gated Linear Recurrent Unit (RG-LRU) is a diagonal linear recurrence h_t = a_t * h_{t-1} + sqrt(1 - a_t^2) * (i_t * x_t) with a recurrence gate a_t and input gate i_t computed from the input — a gated RNN with no softmax and no state expansion. Hawk stacks RG-LRU residual blocks alone; Griffin interleaves two RG-LRU blocks with one local sliding-window attention block, so the recurrence carries global context while the window provides exact recent-token recall with a bounded KV cache. Inference memory is fixed (state + small window) regardless of context length, and the models matched much more token-hungry Transformer baselines at up to 14B scale.

Mostly-Mamba stacks with a few attention layers kept for exact recall — linear-cost long context that still passes needle tests.

AI21's production hybrid: 1 attention layer per 8, MoE on alternate layers; 256K context with a ~8x smaller KV cache, fitting a single 80GB GPU at up to ~140K-token inputs. Zyphra's Mamba tower with ONE shared attention block re-applied every ~6 layers — repeated global attention for the parameter cost of one block. Microsoft's strictly-linear hybrid: Mamba -> MLP -> sliding-window attention -> MLP per unit; trained at 4K, streams past 1M tokens. Mamba-2-heavy hybrids (~92% of layers) at 8B-56B: ~3x faster inference at similar accuracy; Nano 2 tunes the recipe for reasoning. 9 Mamba-2 blocks per 1 attention block, no positional encodings, MoE variants; IBM reports >70% serving-memory reduction. Apache 2.0. Falcon-H1 instead runs attention and Mamba-2 heads in parallel inside every block.

2025's dominant hybrid recipe: 3 gated-DeltaNet linear layers per 1 attention layer; delta-rule state edits plus Mamba-2-style decay.

Gated DeltaNet (NVIDIA/MIT, 2024) combines the delta rule — a rank-1 targeted edit that overwrites the old value stored under a key in the fixed-size matrix state, S_t = a_t * S_{t-1}(I - b_t k_t k_t^T) + b_t v_t k_t^T — with Mamba-2-style global decay gating, beating both Mamba-2 and plain DeltaNet. Qwen3-Next 80B-A3B interleaves three Gated DeltaNet layers per one gated full-attention layer with an ultra-sparse MoE (3B active), reporting roughly 10x throughput versus Qwen3-32B beyond 32K context; Kimi Linear (Moonshot, 2025) refines the operator into Kimi Delta Attention (finer channel-wise decay, chunked kernels) at a 3:1 KDA:MLA ratio in a 48B-A3B model, cutting KV cache by up to 75%. Via the SSD duality these are siblings of Mamba-2 — the SSM and linear-attention branches have effectively merged.

Moonshot AI's 2.78T-parameter sparse-MoE flagship — open weights since 2026-07-27, natively multimodal, 1M context. Every headline number below is now config-verified.

Kimi K3 shipped its open weights on 2026-07-27, so the architecture is no longer announcement-level: it can be read straight from config.json. The released checkpoint is a 93-layer, 2,779,931,837,184-parameter (2.78T) model — the safetensors metadata shows ~2.72T of those params stored as U8, i.e. it ships natively 8-bit quantized (the card is tagged compressed-tensors / 8-bit). It is not text-only: the architecture class is KimiK3ForConditionalGeneration with a vision tower (patchmergerv2 projector, mm_hidden_size 1024, 2×2 patch merging) whose position embeddings carry a time axis — so image and video in, text out (pipeline tag image-text-to-text). The MoE is exactly what was reported and is now confirmed: num_experts 896 with num_experts_per_token 16, plus num_shared_experts 2 (a figure Moonshot never published), sigmoid router scoring with aux-loss-free noaux_tc top-k selection and latent_moe_use_norm — the "Stable LatentMoE" branding. Attention is the KDA hybrid, and the config settles the ratio arithmetically: linear_attn_config lists 69 KDA layers against 24 full-attention layers placed at every 4th position (4, 8, 12 … 92), i.e. 2.88:1 — the 3:1 pattern Moonshot showed in its blog diagram. KDA runs 96 heads at head_dim 128 with a 4-wide short convolution and a full-rank gate; the full-attention layers are MLA (kv_lora_rank 512, q_lora_rank 1536, 96 heads, 128 NoPE + 64 RoPE dims). Attention Residuals are real and parameterized: attn_res_block_size 12. Context is 1,048,576 tokens, vocab 163,840. Note the released config sets num_nextn_predict_layers 0 — no multi-token-prediction head in this checkpoint.

Z.ai's 753B-A40B MIT MoE flagship for long-horizon coding/agents: DSA sparse attention + IndexShare indexer reuse on a solid 1M-token context.

GLM-5.2 is a 78-layer `glm_moe_dsa` transformer: each layer pairs MLA (q_lora_rank 2048 / kv_lora_rank 512 latents, 64 heads, 256-dim QK = 192 NoPE + 64 decoupled RoPE) with a DeepSeek-Sparse-Attention core where a lightweight "lightning indexer" (32 heads x 128 dim) scores past tokens and core attention runs only over the top-2048 (index_topk), cutting attention from O(L^2) to O(Lk). The DSA indexer itself stays O(L^2) and in GLM-5/5.1 ran at every layer, so IndexShare (the linked paper is titled "IndexCache", arXiv:2603.12201) runs real indexers only in "full" layers — layers 0-2 plus every 4th (index_topk_freq=4; 21 of 78) — while the 57 "shared" layers reuse the nearest full layer's top-k indices; in the paper's training-aware variant, retained indexers are distilled against the averaged attention distributions of the layers they serve (the likely deployed recipe, though Z.ai doesn't say outright — see undisclosed). Z.ai claims 2.9x per-token FLOPs reduction at 1M context (vendor-only figure, HF card + NIM); the paper independently measured up to 1.82x prefill / 1.48x decode and 75% indexer-compute removal on a 30B DSA model. The FFN side is DeepSeek-V3-style MoE: 3 dense layers then 75 MoE layers with 256 routed experts (8 active) + 1 shared expert, sigmoid scoring with aux-loss-free `noaux_tc` routing — 753B total all-in; the GLM-5 report's 744B figure covers the main model incl. embeddings/output head but excl. the ~9.9B MTP layer (convention derived from config-param math — the report doesn't state it), 40B active (active figure stated for GLM-5, whose config shape is identical). One MTP layer (num_nextn_predict_layers=1; the GLM-5 report trains 3 MTP layers sharing one parameter set) drives speculative decoding and also reuses the shared indices (`index_share_for_mtp_iteration: true`); Z.ai claims up to 20% longer acceptance length vs 5.1. Context is 1,048,576 positions (rope_theta 8M, up from 202,752 / theta 1M in GLM-5/5.1) with 128K max output on the API; "thinking effort" is purely an inference-time API control (`reasoning_effort`), not an architecture mechanism.

Bolt a pretrained ViT onto an LLM with a small trained projector; image features enter the prompt as soft tokens.

A contrastively pretrained ViT (CLIP or SigLIP) encodes the image into patch features; a small connector — a linear layer or 2-layer MLP in LLaVA, a single-layer cross-attention resampler compressing to 256 tokens in Qwen-VL — maps them into the LLM's embedding space, where they are spliced into the token sequence like ordinary word embeddings. Training is staged: first align only the projector on image-caption pairs, then visual instruction-tune with the LLM (and often later the ViT) unfrozen. Modern variants handle native/dynamic resolution via tiling (InternVL 1.5) or M-RoPE with a ViT trained alongside the LLM (Qwen2-VL), and compress patches (2x2 token merge, pixel-shuffle) to keep image token counts manageable.

Vision enters through gated cross-attention layers inserted between frozen LM blocks — Flamingo's recipe, revived by Llama 3.2 Vision.

A frozen vision encoder feeds a Perceiver Resampler that compresses any number of images or video frames into a fixed set of 64 latent tokens (Flamingo). Newly inserted gated cross-attention-dense layers, interleaved between the frozen LM's blocks, read those latents as keys/values; a tanh gate initialized at zero means the model starts out exactly equal to the original text LM and learns to open the visual pathway during training. Llama 3.2 Vision (disclosed in The Llama 3 Herd of Models) applies the same compositional idea: a ViT encoder plus cross-attention layers inserted every 4th decoder layer, with the text weights kept frozen so text-only performance is provably unchanged. Image information never occupies positions in the token sequence.

One transformer, one stream: images (and audio) become tokens or patches in the same sequence from pretraining day one.

Instead of grafting vision onto a text model, one transformer is trained from the start on interleaved multimodal sequences. Chameleon VQ-quantizes each 512x512 image into 1024 discrete tokens from an 8192-entry codebook and trains a single autoregressive model over mixed text+image tokens — requiring QK-norm and revised norm placement for stability at scale — so the same softmax can emit words or pixels. Fuyu goes further and deletes the encoder entirely: raw image patches are linearly projected straight into the decoder's first layer at arbitrary resolution, with an image-newline token marking row breaks. GPT-4o and Gemini are disclosed only at the capability level — a single model trained end-to-end (OpenAI) / jointly from the start (Google) across text, vision, and audio — with internal mechanics undisclosed.

Omni models take speech, audio, image, video, and text in — and stream speech out — replacing the ASR → LLM → TTS pipeline.

GPT-4o's system card discloses a single autoregressive model that accepts any mix of text, audio, image, and video and emits text, audio, and image from the same network, collapsing voice latency to roughly 300ms versus the old three-model ASR→LLM→TTS pipeline; internals are undisclosed. Qwen2.5-Omni discloses a full open recipe: a Thinker (LLM decoder) ingests all modalities — audio and video time-aligned by TMRoPE, a time-interleaved multimodal RoPE — and produces text plus high-level hidden states, while a Talker, a dual-track autoregressive speech-token model, streams speech tokens conditioned on those states through a sliding-window DiT vocoder for real-time audio. Moshi runs full-duplex: one backbone (Helium 7B) jointly models the user's and the system's audio streams via the Mimi neural codec plus an inner-monologue text stream, reaching ~160ms theoretical latency.

Text generated by iteratively denoising/unmasking whole blocks in parallel, instead of predicting one token at a time.

Instead of left-to-right next-token prediction, the model is trained on a forward corruption process (randomly masking or noising tokens) and learns the reverse process: given a partially masked sequence, predict all masked tokens simultaneously. Generation starts from a fully masked block and runs a few refinement steps, each one committing the most confident tokens and re-predicting the rest. The backbone is still a standard Transformer, but with bidirectional (non-causal) attention, and speed comes from emitting many tokens per forward pass. LLaDA showed this trained-from-scratch at 8B rivals autoregressive peers; Mercury and Gemini Diffusion showed commercial-grade throughput (1,000+ tokens/sec).

Transformers trained from scratch with ternary {-1,0,+1} weights, so matmuls become adds — ~10x less memory and energy.

BitNet replaces every nn.Linear in the Transformer with a BitLinear layer whose weights are constrained to {-1, 0, +1} (~1.58 bits) during training itself — quantization is native, not applied after the fact, so the model learns around the constraint (a straight-through estimator passes gradients while latent full-precision weights are kept for the optimizer). Activations are quantized to 8 bits. Because weights are ternary, matrix multiplication reduces to integer additions and sign flips, slashing memory, energy, and latency. The 2024 paper showed ternary matches FP16 quality from ~3B params; Microsoft's open BitNet b1.58 2B4T (2B params, 4T tokens) ships with a CPU-friendly runtime at ~0.4 GB memory.

o1/R1-class models are the same transformer architecture; the novelty is RL training that elicits long private chain-of-thought.

Honest note for the field guide: despite the branding, o1/R1-style 'reasoning models' are not a new neural architecture. The DeepSeek-R1 paper is explicit that R1 is DeepSeek-V3-Base — a standard MoE transformer (671B total / 37B active) — post-trained with reinforcement learning (GRPO) using rule-based rewards, which spontaneously incentivizes long chain-of-thought, self-verification, and backtracking (R1-Zero used pure RL, R1 added a small cold-start SFT stage). OpenAI's o1 System Card likewise describes large-scale RL to 'think before answering' via chain-of-thought, disclosing nothing about a changed architecture. What changes at inference is behavior and compute allocation (many more generated thinking tokens), not the network.

Sub-3B models engineered for phones: at small scale, deep-and-thin beats wide-shallow, and data curation does the heavy lifting.

MobileLLM (Meta) systematically studied sub-billion-parameter design and found the opposite of conventional wisdom at that scale: allocating parameters to depth (more, thinner layers) beats width, and combining that with embedding sharing, grouped-query attention, and immediate block-wise layer sharing (reusing a block twice before moving on) yields 2-4+ point accuracy jumps over prior 125M/350M models with no extra memory. The SmolLM line (Hugging Face) attacks the same target from the data side: SmolLM2-1.7B is a fairly standard architecture trained on ~11T tokens with multi-stage rebalancing of curated datasets (FineMath, Stack-Edu, SmolTalk), showing meticulous data-centric training makes small models punch far above their weight. SmolLM3-3B adds GQA plus NoPE-style positional handling and dual-mode (think/no-think) reasoning at 3B.

One shared stack of layers applied recursively, with a router giving each token its own number of passes through it.

MoR unifies two efficiency ideas in one recursive Transformer: parameter sharing and adaptive compute. A single block of layers is reused up to N times (recursion), so the parameter count stays small; a lightweight router then decides per token how many recursion steps that token gets — easy tokens exit after one pass, hard tokens loop more, in the spirit of mixture-of-experts routing but over depth. Recursion-wise KV caching keeps attention cost consistent with each token's actual depth. At 135M-1.7B scale it forms a new Pareto frontier over vanilla and prior recursive baselines at equal training FLOPs, with higher inference throughput.

Tokenizer-free LLM over raw bytes: entropy-based dynamic patches feed a big latent transformer; matches Llama 3 with fewer FLOPs.

BLT (Meta) removes the tokenizer entirely and operates on raw bytes, solving the compute problem with a three-part design: a small local encoder groups bytes into dynamically sized patches — patch boundaries are placed where a tiny byte-LM's next-byte entropy is high, so predictable stretches get long patches and hard spots get short ones — a large latent transformer runs only over patch representations, and a small local decoder maps latent outputs back to bytes. Because patches average longer than BPE tokens where text is easy, compute is allocated to where prediction is genuinely hard. Scaled to 8B parameters and 4T training bytes, BLT matches tokenization-based Llama-3-style training while cutting inference FLOPs up to ~50%, and is markedly more robust to spelling noise, character-level tasks, and rare scripts.