Learn AI Create the Future — with Pranay
Main site

CNN Backbones and Pretraining

Inception and GoogLeNet

Inception runs several filter sizes side by side on the same input and glues the results together, and it uses 1x1 convolutions to make that affordable.

Read these first

On this page 7
  1. What was broken before
  2. The problem this creates
  3. What a one-by-one filter even means
  4. Where you have already seen this
  5. The honest part
  6. Remember this
  7. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

An Inception block looks at the same patch with several filter sizes at once. Then it glues all the answers together.

Think about buying a watermelon. You do not use one look. You take in the whole fruit, then a hand-sized patch of skin, then the stem.

Three scales, all at once, and you combine them into one judgement. Nobody told you which scale would settle it, so you use all three.

An Inception block does exactly that with a picture. Small filter, medium filter, large filter, all on the same input, results stacked side by side.

What was broken before

Before this, designing a network meant guessing. Should this layer use a small filter or a large one? Should it pool here or not?

You guessed, trained for a week, and found out. Then you guessed again.

The Inception answer was to stop guessing. Run all the options in parallel and let training decide which paths matter. The useful ones get stronger weights.

The problem this creates

Running four paths costs four times as much. Large filters are the worst offenders. A filter covering a five-by-five patch across hundreds of channels holds an enormous number of weights.

The fix is a bottleneck. It is a tiny layer that squeezes the channel count down, before the expensive filter runs.

   expensive:   192 channels  ->  [ 5x5 filter ]  ->  32 channels

   cheap:       192 channels  ->  [ 1x1 squeeze ] ->  16 channels
                                        |
                                   [ 5x5 filter ] ->  32 channels

A courier does the same thing. Squash the parcel first, then pay for shipping. In the measured example below, this change cuts that path from about 153,000 numbers to about 16,000.

What a one-by-one filter even means

A filter that looks at a single pixel sounds pointless. It is not, because a pixel in the middle of a network is not a colour. It is a stack of hundreds of numbers, one per channel.

A one-by-one filter mixes those channels together at each position. Think of it as blending, not zooming. It changes what is described at each spot without changing where anything is.

Where you have already seen this

  • GoogLeNet won the biggest image recognition competition of 2014, which is when this design became famous.
  • Google Photos and Google Lens grew out of this line of work.
  • The bottleneck idea it introduced is now inside almost every network on your phone.

The honest part

Inception blocks are fiddly. Each one has six numbers to choose. The paper's choices look arbitrary, because they were tuned by hand.

Later architectures replaced this hand-tuning with a repeated, uniform block. Inception is still worth understanding, because the bottleneck it introduced survived everything that followed it.

Remember this

  • An Inception block runs several filter sizes in parallel and joins the outputs.
  • A one-by-one convolution mixes channels and costs almost nothing.
  • Squeezing channels before an expensive filter is the cheapest saving in network design.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install "torch==2.5.1" "torchvision==0.20.1"

Run against PyTorch 2.5.1 and torchvision 0.20.1 on CPU. No weights are downloaded.

Build both versions and count the difference

inception_block.py
import torch
import torch.nn as nn


class NaiveInception(nn.Module):
    """Four paths, no bottlenecks: the version the paper calls too expensive."""
    def __init__(self, c_in, n1, n3, n5, npool):
        super().__init__()
        self.b1 = nn.Conv2d(c_in, n1, 1)
        self.b3 = nn.Conv2d(c_in, n3, 3, padding=1)
        self.b5 = nn.Conv2d(c_in, n5, 5, padding=2)
        self.bp = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1),
                                nn.Conv2d(c_in, npool, 1))

    def forward(self, x):
        return torch.cat([self.b1(x), self.b3(x), self.b5(x), self.bp(x)], dim=1)


class Inception(nn.Module):
    """The real block: a 1x1 convolution shrinks the channels before the big kernels."""
    def __init__(self, c_in, n1, r3, n3, r5, n5, npool):
        super().__init__()
        self.b1 = nn.Conv2d(c_in, n1, 1)
        self.b3 = nn.Sequential(nn.Conv2d(c_in, r3, 1), nn.ReLU(),
                                nn.Conv2d(r3, n3, 3, padding=1))
        self.b5 = nn.Sequential(nn.Conv2d(c_in, r5, 1), nn.ReLU(),
                                nn.Conv2d(r5, n5, 5, padding=2))
        self.bp = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1),
                                nn.Conv2d(c_in, npool, 1))

    def forward(self, x):
        return torch.cat([self.b1(x), self.b3(x), self.b5(x), self.bp(x)], dim=1)


def params(m):
    return sum(p.numel() for p in m.parameters())


# inception_3a from the GoogLeNet paper: 192 channels in, a 28x28 map
naive = NaiveInception(192, 64, 128, 32, 32)
real = Inception(192, 64, 96, 128, 16, 32, 32)

x = torch.zeros(1, 192, 28, 28)
print("input          ", tuple(x.shape))
print("naive output   ", tuple(naive(x).shape))
print("bottleneck out ", tuple(real(x).shape))

print(f"\nparameters, naive block       : {params(naive):>10,}")
print(f"parameters, with 1x1 reduction: {params(real):>10,}")
print(f"saving                        : {params(naive) / params(real):.2f}x")

positions = 28 * 28
print("\nmultiply-accumulates on a 28x28 map")
print(f"  naive      : {params(naive) * positions:>14,}")
print(f"  bottleneck : {params(real) * positions:>14,}")

print("\nwhere the naive cost hides: the 5x5 path alone")
print(f"  192 -> 32 with a 5x5 kernel : {192 * 32 * 25:>10,} weights")
print(f"  192 -> 16 -> 32 in two steps: {192 * 16 + 16 * 32 * 25:>10,} weights")

print("\ncompare with the real GoogLeNet in torchvision")
from torchvision.models import googlenet, vgg16
main = googlenet(weights=None, init_weights=False, aux_logits=False)
with_aux = googlenet(weights=None, init_weights=False, aux_logits=True)
print(f"  its inception3a block : {params(main.inception3a):>12,}")
print(f"  GoogLeNet, main tower : {params(main):>12,}")
print(f"  plus two aux heads    : {params(with_aux):>12,}")
print(f"  VGG-16, same year     : {params(vgg16(weights=None)):>12,}")
Output
input           (1, 192, 28, 28)
naive output    (1, 256, 28, 28)
bottleneck out  (1, 256, 28, 28)

parameters, naive block       :    393,472
parameters, with 1x1 reduction:    163,696
saving                        : 2.40x

multiply-accumulates on a 28x28 map
  naive      :    308,482,048
  bottleneck :    128,337,664

where the naive cost hides: the 5x5 path alone
  192 -> 32 with a 5x5 kernel :    153,600 weights
  192 -> 16 -> 32 in two steps:     15,872 weights

compare with the real GoogLeNet in torchvision
  its inception3a block :      155,872
  GoogLeNet, main tower :    6,624,904
  plus two aux heads    :   13,004,888
  VGG-16, same year     :  138,357,544

Reading the output

Both blocks produce identical shapes. (1, 256, 28, 28) either way. The bottleneck is invisible from outside the block, which is what makes it a drop-in saving.

The whole block gets 2.4 times cheaper. That understates the effect, because the 1x1 and pooling paths were already cheap and dilute the ratio.

The 5x5 path alone drops from 153,600 weights to 15,872, a factor of about ten. Squeezing 192 channels to 16 before the 5x5 kernel is where the entire saving lives. This is the idea that later became the ResNet bottleneck and the MobileNet inverted bottleneck.

torchvision's inception3a reports 155,872 rather than 163,696. The gap is real and explainable. torchvision's BasicConv2d uses bias=False followed by BatchNorm2d, which is not in the 2014 paper. Fewer biases, plus two batch-norm parameters per channel, lands slightly below the hand-written version.

GoogLeNet has 6.6 million parameters against VGG-16's 138 million. Twenty times smaller, in the same competition year, with better accuracy. That comparison is why the Inception paper mattered.

The auxiliary heads double the count and are thrown away. They existed only during training, to inject gradient into the middle of the network. Residual connections made them unnecessary. At inference you use aux_logits=False.

Common mistakes

Concatenating along the wrong dimension. Feature maps are (N, C, H, W), so branches join on dim=1. Using dim=0 stacks along the batch and silently produces four times as many samples.

Branch outputs of different spatial size. Every path must return the same height and width or torch.cat fails. The 3x3 path needs padding=1, the 5x5 path needs padding=2, and the pooling path needs stride=1, padding=1.

Treating a 1x1 convolution as free. It costs c_in * c_out weights per position. Going 192 to 64 channels is 12,288 weights, small next to a 5x5 but not zero. On a 56x56 map, 1x1 convolutions can dominate the compute.

Reaching for Inception in new work. Use it to understand bottlenecks. For a new project, a ResNet or a ConvNeXt of similar size trains more easily and transfers better.

Try it yourself

Change the 5x5 reduction from 16 to 96 and watch the saving evaporate. Then set every reduction equal to the input channel count, and confirm you have rebuilt the naive block with extra steps.

What to learn next

Researcher — Mathematics and papers.

The design argument

Szegedy et al. (2015), Going Deeper with Convolutions, arxiv.org/abs/1409.4842, motivates the block from the theoretical work of Arora et al. on approximating an optimal sparse network structure with dense components. The practical claim is more modest and more useful: correlated units cluster at several scales, so a layer should offer several kernel sizes and let training weight them.

The architecture is 22 layers deep, won ILSVRC 2014 classification, and holds roughly 6.6 million parameters in its main tower. The design constraint stated in the paper is a computational budget held fixed while depth and width increase.

The 1x1 convolution comes from Lin, Chen and Yan (2014), Network In Network, arxiv.org/abs/1312.4400, which also introduced global average pooling in place of large fully connected classifier heads. GoogLeNet adopted both, and the second one is most of the twenty-fold parameter gap against VGG-16.

Cost of the bottleneck

A dense $k \times k$ convolution from $C_{\text{in}}$ to $C_{\text{out}}$ costs

$$ C_{\text{in}} C_{\text{out}} k^2 $$

weights. Inserting a reduction to $R$ channels first gives

$$ C_{\text{in}} R + R\, C_{\text{out}} k^2 . $$

The ratio of the second to the first is

$$ \frac{R}{C_{\text{out}} k^2} + \frac{R}{C_{\text{in}}}, $$

which is small when $R \ll C_{\text{in}}$ and $k$ is large. This is why the 5x5 branch benefits most and the 1x1 branch not at all.

The same algebra recurs throughout this section: in the ResNet bottleneck block, in MobileNet's pointwise convolution, and in transformer attention's projection matrices.

The auxiliary classifiers

Two intermediate heads attached at inception4a and inception4d, weighted 0.3 in the total loss, were introduced to combat vanishing gradients. Szegedy et al. (2015), Rethinking the Inception Architecture, arxiv.org/abs/1512.00567, later reported that they do not accelerate early convergence as claimed, and act mainly as regularisers, particularly when batch normalisation or dropout is added to the head. Residual connections made the mechanism obsolete a year later.

The Inception lineage

VersionYearWhat changed
Inception v1 / GoogLeNet2014Parallel multi-scale branches, 1x1 reductions, global average pooling
Inception v22015Batch normalisation throughout; 5x5 factorised into two 3x3
Inception v32015Asymmetric factorisation of $n \times n$ into $n \times 1$ then $1 \times n$; label smoothing; RMSProp
Inception v4 / Inception-ResNet2016Uniform stem, plus residual connections in the hybrid variants
Xception2017Inception hypothesis pushed to its limit: fully depthwise separable

Rethinking the Inception Architecture reports 21.2% top-1 and 5.6% top-5 error for single-frame evaluation at about 5 billion multiply-adds and under 25 million parameters, with an ensemble reaching 3.5% top-5 on the validation set.

Its factorisation arguments are the durable part. A 5x5 convolution is replaced by two stacked 3x3 convolutions at $18C^2$ weights instead of $25C^2$, and an $n \times n$ convolution is replaced by a $1 \times n$ followed by an $n \times 1$, which is a $\tfrac{2}{n}$ reduction. The paper reports the asymmetric form works poorly on early layers and well on medium-sized feature maps, an empirical caveat that is often dropped when the trick is quoted.

Chollet (2017), Xception, arxiv.org/abs/1610.02357, reframes Inception as a partial decoupling of cross-channel and spatial correlations, and takes the decoupling to completion with depthwise separable convolutions. That step leads directly into the MobileNet family.

Why the design was abandoned

Inception blocks are heavily hand-specified: six width hyper-parameters per block, chosen per stage. This resists scaling rules and neural architecture search alike. ResNet's contribution was partly architectural and partly cultural, in that a uniform repeated block with two knobs, depth and width, is far easier to scale, search over, and reason about. Every backbone after 2016 followed the uniform-block convention.

What to learn next