FID and generative image metrics

Fréchet Inception Distance#

Heusel et al. (2017) model both feature sets as multivariate Gaussians and take the Fréchet (Wasserstein-2) distance between them:

$$ \mathrm{FID} = \lVert \mu_r - \mu_g \rVert_2^2 + \operatorname{Tr}!\left(\Sigma_r + \Sigma_g - 2\left(\Sigma_r \Sigma_g\right)^{1/2}\right) $$

$\mu_r, \Sigma_r$ are the mean and covariance of features from real images, $\mu_g, \Sigma_g$ the same for generated ones. Features are the 2048-dimensional pool3 activations of an Inception-V3 network trained on ImageNet. $(\Sigma_r\Sigma_g)^{1/2}$ is the matrix square root, which is why scipy.linalg.sqrtm appears in every implementation and why numerical rescue terms are standard.

The Gaussian assumption is false. Deep features are neither Gaussian nor unimodal. The estimator is used anyway because the closed form makes it cheap, and because it correlates with human judgement well enough to be useful.

Bias, and why the sample count is part of the metric#

The plug-in estimator of $\Sigma$ from $n$ samples in $d$ dimensions is biased, and the trace term inherits it. Chong and Forsyth (2020), Effectively Unbiased FID and Inception Score, show that $\widehat{\mathrm{FID}}n$ decreases roughly as $1/n$ and propose $\mathrm{FID}\infty$, obtained by computing FID at several sample sizes and extrapolating the linear fit in $1/n$ to zero. The runnable table above reproduces this decay on synthetic data where the true value is exactly zero.

Practical consequence: with $d = 2048$ and $n = 50{,}000$, the covariance estimate uses about 24 samples per dimension. It is a noisy estimate, and the residual bias is not negligible, only conventionalised.

Kernel Inception Distance#

Bińkowski et al. (2018) replace the Gaussian assumption with a kernel two-sample test. For the polynomial kernel $k(x,y) = \left(\frac{1}{d}x^\top y + 1\right)^3$, the unbiased MMD estimator is

$$ \widehat{\mathrm{MMD}}^2 = \frac{1}{m(m-1)}\sum_{i \ne j} k(x_i, x_j) + \frac{1}{m(m-1)}\sum_{i \ne j} k(y_i, y_j) - \frac{2}{m^2}\sum_{i,j} k(x_i, y_j) $$

Excluding the diagonal is what makes it unbiased, and is why it can be negative. KID is therefore usable at small $n$, comes with a standard error for free, and is the right choice when you have a few thousand samples rather than fifty thousand.

Implementation-induced variance#

Parmar et al. (2022), On Aliased Resizing and Surprising Subtleties in GAN Evaluation, document two mechanisms that shift FID independently of image quality:

• Aliased downsampling. Bilinear and bicubic resizing in TensorFlow and PyTorch do not adapt the filter support to the scale factor, so downsampling aliases. Different libraries alias differently, and the resulting FID gap can exceed the gap between competing models.
• JPEG quantisation. Generators cannot reproduce JPEG artefacts. If the reference set was stored as JPEG, saving samples as JPEG at a tuned quality lowers FID without improving anything.

They also note that the Inception weights themselves differ between the original TensorFlow checkpoint and the common PyTorch port, giving a third source of disagreement. clean-fid pins all three.

What FID does not separate#

FID collapses fidelity and diversity into one scalar. Two orthogonal decompositions exist:

Precision and recall for distributions (Sajjadi et al., 2018; refined by Kynkäänniemi et al., 2019). Precision is the fraction of generated samples falling inside the support of the real manifold, estimated with $k$-nearest-neighbour hyperspheres; recall is the fraction of real samples inside the generated manifold's support. A mode-collapsed model has high precision and low recall; an over-smoothed model the reverse. FID cannot tell them apart.

Density and coverage (Naeem et al., 2020) fix the sensitivity of the above to outliers in the $k$-NN radius estimate, and are the better default.

Domain and perceptual alternatives#

• CLIP-FID substitutes CLIP image features for Inception features. Better behaved on non-ImageNet domains, and less sensitive to the resizing pathology.
• CLIPScore (Hessel et al., 2021) measures text-image agreement for conditional generation, which FID does not test at all. A text-to-image model can have excellent FID while ignoring the prompt.
• LPIPS (Zhang et al., 2018) is a paired perceptual distance. It is the right tool for reconstruction, super-resolution and inpainting, where a reference image exists, and the wrong tool for unconditional generation, where none does.
• HYPE (Zhou et al., 2019) and the various two-alternative-forced-choice protocols remain the ground truth for perceptual quality. Every automated metric is a proxy for one of these.

Memorisation#

Near-zero FID is achieved by a model that reproduces its training set. Somepalli et al. (2023) and Carlini et al. (2023) demonstrate verbatim training-image extraction from diffusion models, particularly for images duplicated in the training data. Any generative evaluation should be accompanied by a nearest-neighbour check in feature space against the training set, reported as the fraction of samples within a similarity threshold of a training image.

Papers#

• Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NeurIPS 2017 — arxiv.org/abs/1706.08500
• Bińkowski et al., Demystifying MMD GANs, ICLR 2018 — arxiv.org/abs/1801.01401
• Kynkäänniemi et al., Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019 — arxiv.org/abs/1904.06991
• Chong and Forsyth, Effectively Unbiased FID and Inception Score, CVPR 2020 — arxiv.org/abs/1911.07023
• Naeem et al., Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020 — arxiv.org/abs/2002.09797
• Parmar et al., On Aliased Resizing and Surprising Subtleties in GAN Evaluation, CVPR 2022 — arxiv.org/abs/2104.11222
• Zhang et al., The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (LPIPS), CVPR 2018 — arxiv.org/abs/1801.03924

What to learn next#

• Diffusion models — the generators these numbers are used to compare.
• GANs — where mode collapse, and this metric, came from.
• Testing robustness to corruptions — another case of one number hiding several failure modes.