Learn AI Create the Future — with Pranay
Main site

Inside a Transformer Block

SwiGLU and gated feedforward layers

Compute two things from the token instead of one, and let the first decide how much of the second gets through - the feedforward design nearly every current model uses.

Read these first

On this page 6
  1. What was there before
  2. Why this is better
  3. The catch, and the fix
  4. Where you will see it
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A gated feedforward layer computes two separate things from the token, then multiplies them. One decides how much of the other passes through.

Think of a garden hose with a tap. The hose carries the water. The tap decides how much of it comes out. Turn the tap fully and you get everything. Turn it down and you get a trickle. Close it and you get nothing.

Two independent controls, and the result is their combination. The hose does not decide the flow rate, and the tap does not decide what is in the water.

That is a gate. In a transformer, one set of numbers is the water and another set is the tap.

What was there before

The plain feedforward layer widens the token, squashes it, and narrows it back. The squashing step is fixed: the same rule applied to every value, with nothing to adapt it.

The gated version replaces that fixed rule. It computes the widened token twice, from two separate matrices. One copy is passed through the squashing rule and becomes the tap. The other stays as it is and becomes the water. Then they are multiplied, value by value.

             ┌──► matrix A ──► squash ──┐
   token ───►│                          ├──► multiply ──► matrix C ──► out
             └──► matrix B ─────────────┘
                                    (A is the tap, B is the water)

Why this is better

The old design applied the same fixed rule everywhere. The new design lets the model learn how open the tap should be. That choice is separate for each of thousands of slots, and different for every token.

A slot can now be shut off completely for one kind of token and wide open for another. That is a genuinely different kind of flexibility, and it turns out to help.

The honest caveat is that nobody has a satisfying explanation for why it helps. The paper that introduced these variants ends with a line researchers still quote. It offers no explanation, and attributes their success to divine benevolence. That is a real quotation from a real paper, and it is unusually candid.

The catch, and the fix

Three matrices instead of two means one and a half times the parameters for the same width. That would not be a fair comparison against the old design.

So the fix is to make the middle narrower. Shrink the wide part to two thirds of what it was. The three matrices then total the same as the old two. Same size, same cost, better results.

This is why real model configs have odd-looking numbers. The old rule was four times the model width. Two thirds of four is about two and two thirds. That gets rounded up to something a graphics card likes. That is how a model with width 4,096 ends up with an inner width of 11,008.

Where you will see it

Llama, Mistral, Gemma, Qwen, PaLM and most other recent models use a gated feedforward layer. It is the default in new architectures.

Older models — the original transformer, BERT, GPT-2, GPT-3 — use the plain two-matrix version. If you are reading code from before about 2021, expect the old form.

Remember this

  • Two widened copies of the token: one becomes a tap, the other is what flows.
  • The tap is learned per slot and adapts to each token, which the old fixed rule could not do.
  • Shrink the middle to two thirds so the parameter count stays the same as before.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch

Both designs, with the parameter accounting made explicit

swiglu.py
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)
torch.set_printoptions(precision=3, sci_mode=False)

class FFN(nn.Module):
    """The 2017 design: expand, squash, shrink."""
    def __init__(self, d, hidden):
        super().__init__()
        self.up, self.down = nn.Linear(d, hidden, bias=False), nn.Linear(hidden, d, bias=False)
    def forward(self, x):
        return self.down(F.relu(self.up(x)))

class SwiGLU(nn.Module):
    """Two expansions. One is squashed, the other multiplies it."""
    def __init__(self, d, hidden):
        super().__init__()
        self.gate = nn.Linear(d, hidden, bias=False)
        self.up   = nn.Linear(d, hidden, bias=False)
        self.down = nn.Linear(hidden, d, bias=False)
    def forward(self, x):
        return self.down(F.silu(self.gate(x)) * self.up(x))   # elementwise multiply

d = 512
print("keeping the parameter count matched needs the 2/3 rule")
print(f"{'design':>10} {'hidden':>8} {'matrices':>9} {'parameters':>12}")
plain = FFN(d, 4 * d)
naive = SwiGLU(d, 4 * d)
fair  = SwiGLU(d, int(4 * d * 2 / 3))
for name, m, h in (("FFN", plain, 4*d), ("SwiGLU", naive, 4*d), ("SwiGLU", fair, int(4*d*2/3))):
    print(f"{name:>10} {h:>8} {len(list(m.parameters())):>9} "
          f"{sum(p.numel() for p in m.parameters()):>12,}")
print("  -> three matrices instead of two, so shrink the hidden width by 2/3")

print("\nwhat real models round it to (multiple of 256 keeps GPU kernels happy):")
for name, dm, actual in (("Llama 3 8B", 4096, 14336), ("Llama 2 7B", 4096, 11008)):
    two_thirds = int(8 * dm / 3)
    rounded = 256 * ((two_thirds + 255) // 256)
    print(f"  {name:12s} d_model={dm}  8/3*d={two_thirds}  rounded={rounded}  published={actual}")

print("\nwhat the gate actually does, on one 6-wide hidden layer")
g = torch.tensor([[-4., -1., 0., 1., 4., 8.]])
u = torch.tensor([[ 5.,  5., 5., 5., 5., 5.]])
print("  gate values      ", g[0])
print("  silu(gate)       ", F.silu(g)[0])
print("  up values        ", u[0])
print("  what goes forward", (F.silu(g) * u)[0])
print("  -> a strongly negative gate closes that channel to almost nothing")
print("  -> a strongly positive gate lets it through nearly untouched")

x = torch.randn(2, 7, d)
print("\nboth designs are drop-in replacements:")
print("  FFN   ", tuple(plain(x).shape))
print("  SwiGLU", tuple(fair(x).shape))
Output
keeping the parameter count matched needs the 2/3 rule
    design   hidden  matrices   parameters
       FFN     2048         2    2,097,152
    SwiGLU     2048         3    3,145,728
    SwiGLU     1365         3    2,096,640
  -> three matrices instead of two, so shrink the hidden width by 2/3

what real models round it to (multiple of 256 keeps GPU kernels happy):
  Llama 3 8B   d_model=4096  8/3*d=10922  rounded=11008  published=14336
  Llama 2 7B   d_model=4096  8/3*d=10922  rounded=11008  published=11008

what the gate actually does, on one 6-wide hidden layer
  gate values       tensor([-4., -1.,  0.,  1.,  4.,  8.])
  silu(gate)        tensor([-0.072, -0.269,  0.000,  0.731,  3.928,  7.997])
  up values         tensor([5., 5., 5., 5., 5., 5.])
  what goes forward tensor([-0.360, -1.345,  0.000,  3.655, 19.640, 39.987])
  -> a strongly negative gate closes that channel to almost nothing
  -> a strongly positive gate lets it through nearly untouched

both designs are drop-in replacements:
  FFN    (2, 7, 512)
  SwiGLU (2, 7, 512)

The Llama rows are the most instructive line here

Llama 2 7B matches the rule exactly: 11,008. Eight thirds of 4,096 is 10,922, rounded up to the next multiple of 256 gives 11,008. That is the published value. The rule is real and was followed.

Llama 3 8B does not: it uses 14,336. That is 3.5 times the model width, not 2.67. Same width, same head count, and a deliberately larger feedforward layer.

Take that seriously rather than treating the 2/3 rule as a law. It is the setting that makes a fair comparison against a 4x plain feedforward layer. Once you stop comparing and start designing, the ratio is a free choice. Llama 3 spent more of its budget there.

The gate, read carefully

The silu(gate) row shows what SiLU does. -4 becomes -0.072, effectively closed. +8 becomes +7.997, effectively open. 0 becomes exactly 0.

Notice that a negative gate does not close the channel — it inverts it, weakly. -1 maps to -0.269, so the value passes through with its sign flipped and shrunk. That is real capacity, and it is one thing a plain ReLU cannot do.

Notice also that SiLU is unbounded above, so silu(8) * 5 gives 39.99. Gated layers can produce large activations. That is one reason mixed-precision training pairs them with careful normalisation.

The family

All of these are the same construction with a different squashing function on the gate:

NameGate functionUsed in
GLUsigmoidthe original, Dauphin et al. 2017
ReGLUReLUrarely, in ablations
GEGLUGELUT5 v1.1, some vision models
SwiGLUSiLU / SwishLlama, Mistral, Gemma, PaLM

Shazeer's 2020 paper tested all of them. The gated variants beat plain ReLU and GELU baselines on the same budget. GEGLU and SwiGLU were the strongest, and the gap between them was small.

Common mistakes

Keeping the 4x multiplier when you switch to gating. You have quietly increased the layer by fifty percent, and any comparison you make afterwards is invalid.

Adding biases. Every published SwiGLU implementation uses bias=False on all three matrices. Adding them back changes the parameter count and matches nothing.

Squashing the wrong branch. The activation goes on the gate, and the up branch stays linear. Swapping them gives a different function. This mistake produces a model that trains and quietly underperforms.

Loading a plain-FFN checkpoint into a gated module. The shapes will not match, which is fortunate. Loading a SwiGLU checkpoint with the gate and up matrices swapped will match, and will produce nonsense. Check the parameter names against the reference implementation.

Try it yourself

Set self.gate and self.up to the same weights. Confirm the layer becomes down(silu(h) * h), a self-gated activation with no independent tap. Then measure how much of the gate's output is negative on random inputs. Roughly half will be. That is the capacity the plain ReLU version throws away.

What to learn next

Researcher — Mathematics and papers.

Definition

Dauphin et al. (2017), Language Modeling with Gated Convolutional Networks, arXiv:1612.08083, introduced the gated linear unit:

$$ \operatorname{GLU}(x, W, V) = \sigma(xW) \otimes xV $$

Shazeer (2020), arXiv:2002.05202, substitutes other functions for $\sigma$. The result goes in the transformer feedforward position:

$$ \operatorname{FFN}_{\text{SwiGLU}}(x, W, V, W_2) = \left( \operatorname{Swish}_1(xW) \otimes xV \right) W_2 $$

with $\operatorname{Swish}_\beta(x) = x \sigma(\beta x)$ and $\beta = 1$, which is the same function PyTorch calls silu. The paper's baseline is

$$ \operatorname{FFN}_{\text{ReLU}}(x, W_1, W_2) = \max(xW_1, 0) W_2 $$

On the parameter matching, the paper is explicit:

To keep the number of parameters and the amount of computation constant, we reduce the number of hidden units $d_{ff}$ by a factor of $2/3$ when comparing these layers to the original two-matrix version.

On mechanism, it is equally explicit:

We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.

Cost

DesignMatricesParametersFLOPs per token
Plain, $d_{\text{ff}} = 4d$2$8d^2$$16d^2$
Gated, $d_{\text{ff}} = 4d$3$12d^2$$24d^2$
Gated, $d_{\text{ff}} = \tfrac{8}{3}d$3$8d^2$$16d^2$

The matched-budget row is the fair comparison and the one Shazeer reports. Arithmetic intensity differs even at matched FLOPs. Three narrower matmuls plus an elementwise product use hardware worse than two wider matmuls. Matched FLOPs does not mean matched wall-clock. Production implementations fuse the gate and up projections into one matrix multiply of width $2 d_{\text{ff}}$. Splitting the result afterwards recovers most of the difference.

What ratios real models use

Model$d$$d_{\text{ff}}$Ratio
Llama 2 7B4096110082.69
Llama 3 8B4096143363.50

Llama 2 follows $\lceil \tfrac{8}{3} d \rceil$ rounded to a multiple of 256. Llama 3 does not, at the same width. The $2/3$ factor is a device for holding a comparison fixed, not a scaling law. Treating it as one is a common misreading.

Evidence quality

Shazeer's results are on T5-style encoder-decoder pretraining with matched budgets, reporting improvements on GLUE, SuperGLUE and SQuAD. Narang et al. (2021), arXiv:2102.11972, re-evaluated dozens of published transformer modifications under one codebase. Most did not transfer. Gated feedforward variants were among the few that did. That replication is the strongest reason to take SwiGLU seriously. It is stronger evidence than the original paper alone.

No mechanistic account has since become standard. Several explanations circulate: multiplicative interactions increasing expressivity, the gate as learned per-channel conditional computation, better-conditioned gradients. All are plausible, and none is well tested.

Relationship to conditional computation

The gate is a soft, dense selection over hidden units. Mixture-of-experts is a hard, sparse selection over whole feedforward blocks. See Shazeer et al. (2017), arXiv:1701.06538, and Fedus et al. (2021), arXiv:2101.03961. Both introduce input-dependent multiplicative routing; they differ in granularity and in whether the unselected computation is actually skipped. Modern sparse models routinely use SwiGLU inside each expert, so the two compose rather than compete.

Papers

What to learn next