Image-to-image translation

Paired translation#

Isola et al. (2017), Image-to-Image Translation with Conditional Adversarial Networks, formulate the task as a conditional GAN:

$$ \mathcal{L}{cGAN}(G, D) = \mathbb{E}{x,y}[\log D(x, y)] + \mathbb{E}_{x,z}[\log(1 - D(x, G(x, z)))] $$

$x$ is the input image, $y$ the paired target, $z$ a noise vector. Both $G$ and $D$ are conditioned on $x$, which is what distinguishes this from an unconditional GAN. The full objective adds a reconstruction term:

$$ G^* = \arg\min_G \max_D \; \mathcal{L}{cGAN}(G, D) + \lambda \,\mathbb{E}{x,y,z}\big[|y - G(x,z)|_1\big], \qquad \lambda = 100 $$

Two design findings from the paper are worth stating precisely.

The noise $z$ is nearly ignored. The authors report that the generator learns to disregard added noise. They supply stochasticity only through dropout at test time. The output remains close to deterministic. Paired translation with an $L1$ term is effectively a one-to-one map, which matters when the true mapping is one-to-many.

$L1$ is chosen over $L2$ deliberately. Both encourage blur, since the minimiser of an expected pixel loss is a conditional mean, but $L1$ blurs less. The adversarial term then supplies the high frequencies the reconstruction loss cannot.

The generator is a U-Net with skip connections between mirrored layers. For translation tasks the input and output share structure, and the skips let low-level information bypass the bottleneck.

Unpaired translation#

Zhu et al. (2017), Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, learn $G: X \to Y$ and $F: Y \to X$ with two discriminators and:

$$ \mathcal{L}{cyc}(G, F) = \mathbb{E}{x}\big[|F(G(x)) - x|1\big] + \mathbb{E}{y}\big[|G(F(y)) - y|_1\big] $$

$$ \mathcal{L} = \mathcal{L}_{GAN}(G, D_Y, X, Y) + \mathcal{L}_{GAN}(F, D_X, Y, X) + \lambda \mathcal{L}_{cyc}(G, F), \qquad \lambda = 10 $$

The adversarial losses use the least-squares form of Mao et al. (2017) rather than log-likelihood, which the authors report as more stable.

The identifiability problem is real and acknowledged in the paper. With sufficient capacity, $G$ can map $X$ to any permutation of $Y$ matching the target distribution. And $F$ can invert it. Cycle consistency constrains the composition $F \circ G$ to the identity, not $G$ itself. The prior that makes CycleGAN work in practice is architectural and optimisational, not a property of the objective.

Chu et al. (2017), CycleGAN, a Master of Steganography (NIPS 2017 Workshop on Machine Deception), demonstrate the sharp version of this. The generator embeds a nearly imperceptible high-frequency encoding of the source into its output, which the reverse generator decodes. They connect the phenomenon to adversarial examples and argue the cycle loss makes CycleGAN unusually vulnerable to adversarial attack. The practical consequence: cycle reconstruction error is not evidence of semantic correctness.

Beyond the two originals#

CUT (Park et al., 2020) is the most important entry for practitioners moving off CycleGAN. It maximises mutual information between corresponding input and output patches, using a PatchNCE loss. That eliminates the reverse generator, halves memory, and removes the steganography channel by construction.

Palette (Saharia et al., 2022) marks the handover to diffusion. A single conditional diffusion model, with no task-specific architecture and no adversarial loss. It matches or beats specialised GANs on colourisation, inpainting, uncropping and JPEG restoration. It also restores genuine one-to-many behaviour, which pix2pix had lost.

Evaluation#

Pixel metrics are close to meaningless here, because many outputs are correct. The standard battery:

• FID against the target domain, for realism and coverage.
• A pretrained task network on the output, the "FCN-score" of Isola et al. Translate a semantic map to a photo, then run a segmentation network on the result. Check that it recovers the map. This measures whether content survived, and is the closest thing to an objective correctness test for unpaired translation.
• Human study with a two-alternative forced choice against real images.

The FCN-score idea generalises well. Suppose you have an off-the-shelf model that reads the content you claim to preserve. Run it on the output and report the number.

Papers#

• Isola et al., Image-to-Image Translation with Conditional Adversarial Networks, 2017 — arxiv.org/abs/1611.07004
• Zhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, 2017 — arxiv.org/abs/1703.10593
• Chu et al., CycleGAN, a Master of Steganography, 2017 — arxiv.org/abs/1712.02950
• Huang et al., Multimodal Unsupervised Image-to-Image Translation (MUNIT), 2018 — arxiv.org/abs/1804.04732
• Park et al., Semantic Image Synthesis with Spatially-Adaptive Normalization (SPADE), 2019 — arxiv.org/abs/1903.07291
• Park et al., Contrastive Learning for Unpaired Image-to-Image Translation (CUT), 2020 — arxiv.org/abs/2007.15651
• Saharia et al., Palette: Image-to-Image Diffusion Models, 2022 — arxiv.org/abs/2111.05826
• Brooks et al., InstructPix2Pix, 2023 — arxiv.org/abs/2211.09800

What to learn next#

• The denoising U-Net — the network that took over this job from GANs.
• Image segmentation — where the U-Net architecture came from.
• Diffusion models — the generative family behind Palette and InstructPix2Pix.