Image formats and compression artefacts

JPEG throws away colour detail and re-encoding compounds the damage, so the pixels your model sees in production are rarely the pixels the camera captured.

By Pranay Mahendrakar
16 min read
3 reading levels
Updated 31 Aug 2026

Read these first

On this page 10

The short answer
The analogy you have lived
Two kinds of format
The part that surprises everyone
What the damage looks like
The other one: the sideways photo
Where you have seen this
The honest part
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

JPEG makes files smaller by throwing detail away. The detail it discards first is colour, which many models rely on.

The analogy you have lived

Someone photocopies a document and hands you the copy. It looks fine.

Then somebody photocopies your copy. Then a third person copies that one. By the fifth generation the small print is a grey smear.

No single copy destroyed the page. Each one lost a little, and the losses stacked up. Digital images do exactly this. Unlike the photocopier, nothing about the file says how many times it happened.

Two kinds of format

Lossless formats — PNG, and TIFF in most settings — store every pixel exactly. Save and reload a thousand times and the numbers never change. The files are bigger.

Lossy formats — JPEG, and WebP in its usual mode — throw information away. The file gets much smaller. Save and reload once, and the pixels have already changed.

Almost every photo that reaches a production system is a JPEG. Phones produce them, browsers upload them, and every service between compresses them again.

The part that surprises everyone

You would expect the quality setting to control how much is lost. It does, partly. But there is a second setting, usually invisible, that costs more than the quality setting does.

JPEG stores brightness at full detail and colour at half detail by default. It halves the colour resolution in both directions. Three quarters of the colour is gone before the quality setting is considered.

This is a very good trade for human eyes, which are far more sensitive to brightness than to colour. It is a bad trade wherever colour carries the meaning. A red warning light against an amber one. A coloured wire in an equipment photo. A chart with thin coloured lines.

What the damage looks like

Compression works on small square blocks. When it runs out of budget, block edges start to show, and sharp boundaries grow coloured fringes.

   original edge          after heavy compression
   RRRRR|BBBBB            RRRRR|BBBBB
   RRRRR|BBBBB            RRRPP|PPBBB    <- colour bleeds across
   RRRRR|BBBBB            RRRRR|BBBBB       and blocks appear

A model trained on clean images and served compressed ones is looking at a slightly different world. A model trained on compressed images and served clean ones has the same problem in reverse.

The other one: the sideways photo

Phone cameras do not rotate the picture when you turn the phone. They store the picture as the sensor saw it and add a note saying which way is up.

Some libraries read that note. Some ignore it. Suppose your training code read it and your serving code did not. Half your production images now arrive rotated a quarter turn. Nothing in your logs will say so.

Where you have seen this

A photo that looks upright in your gallery and sideways when you upload it.
Screenshots of screenshots that look progressively worse.
A chat app that makes the photo you send noticeably softer.
A scanned form that goes blotchy after being emailed a few times.

The honest part

You usually cannot control the format your inputs arrive in. Users upload what their phone produced, and their phone already compressed it.

So the aim is not to avoid compression. It is to make sure the compression your model was trained on resembles the compression it will be served. Trained on pristine PNGs and serving phone JPEGs? You have a gap. Close it by compressing your training data, not by hoping.

Remember this

PNG keeps every pixel. JPEG throws detail away, colour detail first.
The default colour-halving costs more quality than dropping the quality setting a long way.
Re-saving a JPEG damages it again, every time, and nothing records that it happened.

What to learn next

Train and serve preprocessing mismatch — the other half of the input-fidelity problem.
Testing robustness to corruptions — measuring what compression costs your accuracy.
How images are stored — the pixel representation underneath all of this.

Developer — Code and libraries.

Setup

bash

pip install numpy pillow

Pillow 11.0.0 was used below. The measurements are deterministic: the same test image and the same encoder settings give the same numbers on any machine.

Measuring what compression costs

format_damage.py

import io
import numpy as np
from PIL import Image, ImageOps

# A 256x256 test picture: smooth sky, a hard red/blue edge, and fine texture.
yy, xx = np.mgrid[0:256, 0:256]
img = np.zeros((256, 256, 3), np.uint8)
img[..., 0] = np.clip(60 + xx * 0.4, 0, 255)
img[..., 1] = np.clip(90 + yy * 0.3, 0, 255)
img[..., 2] = np.clip(200 - yy * 0.2, 0, 255)
img[64:192, 64:128] = [220, 20, 40]                       # a saturated red block
img[64:192, 128:192] = [20, 40, 220]                      # against a blue one
img[200:250, 20:236] = ((xx[200:250, 20:236] % 4 < 2) * 255)[..., None]  # 2px stripes
src = Image.fromarray(img)


def roundtrip(im, **kw):
    buf = io.BytesIO()
    im.save(buf, "JPEG", **kw)
    return np.asarray(Image.open(buf), np.float64), buf.getbuffer().nbytes


def psnr(a, b):
    mse = ((a - b) ** 2).mean()
    return float("inf") if mse == 0 else 10 * np.log10(255.0 ** 2 / mse)


ref = np.asarray(src, np.float64)

print(f"{'JPEG quality':>12} {'bytes':>7} {'PSNR dB':>8} {'max err':>8} {'mean err':>9}")
for q in (100, 95, 85, 75, 60, 40, 20, 10):
    out, n = roundtrip(src, quality=q)
    d = np.abs(out - ref)
    print(f"{q:>12} {n:>7} {psnr(ref, out):>8.2f} {d.max():>8.0f} {d.mean():>9.2f}")

png = io.BytesIO()
src.save(png, "PNG")
back = np.asarray(Image.open(png), np.float64)
print(f"\nPNG: {png.getbuffer().nbytes} bytes, max error {np.abs(back-ref).max():.0f} "
      f"(lossless, so exactly zero)")

print("\nchroma subsampling at quality 90 (the red/blue edge is the interesting part):")
for name, sub in (("4:4:4 (subsampling=0)", 0), ("4:2:2 (subsampling=1)", 1),
                  ("4:2:0 (subsampling=2)", 2)):
    out, n = roundtrip(src, quality=90, subsampling=sub)
    edge = slice(64, 192), slice(120, 136)
    print(f"  {name:<22} {n:>6} bytes   whole image {psnr(ref, out):>5.2f} dB"
          f"   at the edge {psnr(ref[edge], out[edge]):>5.2f} dB")

print("\nquality versus subsampling, head to head:")
for q, sub in ((100, 0), (100, 2), (85, 0), (85, 2)):
    out, n = roundtrip(src, quality=q, subsampling=sub)
    print(f"  quality {q:>3}, subsampling {sub}: {n:>6} bytes, {psnr(ref, out):>5.2f} dB")

print("\nre-saving the same JPEG over and over (quality 85 each time):")
cur = src
for gen in range(1, 11):
    buf = io.BytesIO()
    cur.save(buf, "JPEG", quality=85)
    cur = Image.open(buf)
    if gen in (1, 2, 3, 5, 10):
        arr = np.asarray(cur, np.float64)
        print(f"  generation {gen:>2}: {psnr(ref, arr):>5.2f} dB, "
              f"max error {np.abs(arr-ref).max():>3.0f}")

print("\nEXIF orientation, the bug that silently rotates your input:")
tall = Image.fromarray(img[:128])                  # 256 wide, 128 tall
ex = Image.Exif()
ex[274] = 6                                        # 6 = rotate 90 degrees clockwise
b = io.BytesIO()
tall.save(b, "JPEG", exif=ex)
loaded = Image.open(b)
print(f"  size as decoded            {loaded.size}   orientation tag "
      f"{loaded.getexif().get(274)}")
print(f"  size after exif_transpose  {ImageOps.exif_transpose(loaded).size}")

Output

JPEG quality   bytes  PSNR dB  max err  mean err
         100   15843    35.89       81      0.89
          95    8827    35.85       81      0.94
          85    5934    35.67       81      1.11
          75    4970    35.28       81      1.43
          60    4638    34.19       78      1.86
          40    3957    33.46       80      2.38
          20    3253    31.18       96      3.94
          10    3041    29.55      123      5.03

PNG: 1171 bytes, max error 0 (lossless, so exactly zero)

chroma subsampling at quality 90 (the red/blue edge is the interesting part):
  4:4:4 (subsampling=0)    8256 bytes   whole image 48.89 dB   at the edge 51.14 dB
  4:2:2 (subsampling=1)    7261 bytes   whole image 37.30 dB   at the edge 25.22 dB
  4:2:0 (subsampling=2)    6607 bytes   whole image 35.74 dB   at the edge 25.01 dB

quality versus subsampling, head to head:
  quality 100, subsampling 0:  23322 bytes, 52.54 dB
  quality 100, subsampling 2:  15843 bytes, 35.89 dB
  quality  85, subsampling 0:   7441 bytes, 48.03 dB
  quality  85, subsampling 2:   5934 bytes, 35.67 dB

re-saving the same JPEG over and over (quality 85 each time):
  generation  1: 35.67 dB, max error  81
  generation  2: 33.41 dB, max error  96
  generation  3: 31.96 dB, max error 104
  generation  5: 30.25 dB, max error 114
  generation 10: 28.21 dB, max error 124

EXIF orientation, the bug that silently rotates your input:
  size as decoded            (256, 128)   orientation tag 6
  size after exif_transpose  (128, 256)

The finding that should change how you save images

Quality 100 scores 35.89 dB. Quality 85 scores 35.67 dB. Fifteen quality points and nearly three times the file size bought 0.22 dB. That looks broken, and it is not.

The head-to-head table explains it. Quality 85 with subsampling off scores 48.03 dB in 7,441 bytes. Quality 100 with the default subsampling scores 35.89 dB in 15,843 bytes. The lower-quality, smaller file is 12 dB better.

Pillow's default subsampling=-1 means "let the encoder decide", and libjpeg decides on 4:2:0 — quarter-resolution colour — regardless of the quality setting. Cranking quality to 100 does nothing about it. If your images matter, pass subsampling=0 explicitly and then choose a quality.

The edge column shows where the loss lands. At 4:4:4 the red/blue boundary scores 51.14 dB. At 4:2:0 it scores 25.01 dB, a 26 dB collapse, while the whole-image number falls by only 13 dB. Chroma subsampling damage concentrates exactly at saturated colour boundaries, which is where anything colour-coded lives.

Generation loss is real and it does not stop. Ten re-saves at quality 85 took the image from 35.67 dB to 28.21 dB, with the maximum error growing from 81 to 124. Each save requantises coefficients that were already quantised. There is no equilibrium.

PNG is 1,171 bytes here, smaller than every JPEG in the table. That is a property of this synthetic image — flat gradients and hard edges compress well losslessly. On a real photograph PNG would be several times larger than JPEG. Do not generalise the size, do generalise the zero error.

The EXIF block is the quiet one. The image decodes as 256 by 128 and carries orientation tag 6. ImageOps.exif_transpose turns it into 128 by 256. If your training loader called that and your serving code did not, every portrait photo from a phone reaches your model on its side, and no error is raised.

What to actually do

Compress your training data the way production will compress it. If users upload phone JPEGs, do not train on pristine PNGs. Add a JPEG round-trip at a realistic quality into your augmentation pipeline. It is the cheapest robustness win available.

python

import io, random
from PIL import Image

def jpeg_augment(im, qualities=(95, 85, 75, 60)):
    """Re-encode through JPEG so the model sees the artefacts it will be served."""
    buf = io.BytesIO()
    im.save(buf, "JPEG", quality=random.choice(qualities))
    buf.seek(0)
    return Image.open(buf).convert("RGB")

Decode once, in memory, and pass arrays onward. Every intermediate service that saves a JPEG and re-reads it adds a generation of loss.

Call exif_transpose in exactly one place, and make it the same place for training and serving — ideally inside the preprocessing module you ship with the model.

Store your evaluation set losslessly. Your benchmark should not be re-encoded by whatever tool touched it last.

Common mistakes

Assuming quality 95 is "nearly lossless". It is not, and the default subsampling makes the quality setting the smaller of the two effects.

Using PNG for photographs to be safe. Files five to ten times larger, and your users' images were already JPEG before they reached you. The loss has happened.

Converting to JPEG for storage inside a pipeline. Decode once, keep arrays, or store the original bytes. Never re-encode as an intermediate step.

Trusting the file extension. A file named .png can be a JPEG. Check the magic bytes, or let the decoder tell you Image.format.

Ignoring ICC colour profiles. An image tagged Display P3 and decoded as sRGB has systematically shifted colours. Convert explicitly with ImageCms, or strip and standardise on sRGB, but decide.

Treating WebP and AVIF as lossless by default. Both have lossless modes and both default to lossy.

Try it yourself

Change the fine stripes from two pixels wide to one and rerun the quality table. Predict first whether PSNR goes up or down. Then set subsampling=0 in the generation-loss loop and see how much of the ten-generation decay was chroma damage rather than quality loss.

What to learn next

Train and serve preprocessing mismatch — the other half of the input-fidelity problem.
Testing robustness to corruptions — measuring what compression costs your accuracy.
How images are stored — the pixel representation underneath all of this.

Researcher — Mathematics and papers.

What JPEG actually does

Baseline JPEG (ITU-T T.81, 1992) is a five-stage pipeline:

Colour transform. RGB to $Y'C_bC_r$, concentrating perceptually important information in the luma channel $Y'$.
Chroma subsampling. $C_b$ and $C_r$ are downsampled. 4:2:0 halves both dimensions, discarding 75 percent of chroma samples before any transform coding.
Block DCT. Each channel is split into 8x8 blocks and transformed: $$ F(u,v) = \frac{1}{4}C(u)C(v)\sum_{x=0}^{7}\sum_{y=0}^{7} f(x,y)\cos\frac{(2x+1)u\pi}{16}\cos\frac{(2y+1)v\pi}{16} $$ with $C(0) = 1/\sqrt{2}$ and $C(k) = 1$ otherwise.
Quantisation. $\hat{F}(u,v) = \operatorname{round}!\left(F(u,v)/Q(u,v)\right)$. This is the only lossy step in transform coding, and $Q$ is scaled by the quality setting.
Entropy coding. Huffman or arithmetic coding of the quantised coefficients. Lossless.

The measurements above follow directly. Stage 2 is a fixed, quality-independent loss, which is why quality 100 with 4:2:0 cannot beat quality 85 with 4:4:4. Stage 4 is what the quality slider moves.

Generation loss

Re-encoding is not idempotent in general. The decoded image differs from the original by quantisation error and by the chroma resampling; re-encoding quantises the new coefficients, which are no longer aligned to the previous quantisation lattice. Chroma upsampling and downsampling in particular are not inverses.

There is a special case: re-encoding with an identical quantisation table, identical subsampling and identical 8x8 block alignment is close to idempotent after the first generation. Any crop that is not a multiple of 8 (or 16 with subsampling) destroys the alignment and restarts the decay. This misalignment signature is what JPEG forensics uses to detect edited regions.

Effects on models

Dodge and Karam (2016) and Hendrycks and Dietterich (2019) both measure classification accuracy against JPEG quality; degradation is modest down to roughly quality 50 and then steepens. JPEG is one of the fifteen ImageNet-C corruptions for exactly this reason — see testing robustness to corruptions.

Three consequences that are specific rather than general:

Detection and segmentation degrade earlier than classification. Both depend on precise boundaries, which is where blocking artefacts and chroma bleed concentrate.
Compression is a partial adversarial defence and a partial attack. JPEG removes high-frequency perturbations, which motivated it as a preprocessing defence; it also introduces its own high-frequency structure, and compression-aware attacks defeat the defence.
Compression level is a distribution-shift dimension. A model trained on one platform's re-encoding pipeline and served another's is subject to measurable covariate shift. Log the encoder, quality and subsampling of production inputs; see detecting drift in image data.

PSNR, and why it is a weak instrument

$$ \mathrm{PSNR} = 10\log_{10}\frac{\mathrm{MAX}^2}{\mathrm{MSE}} $$

It is a monotone function of mean squared error and inherits every one of MSE's problems: it is blind to structure, it weights all spatial positions equally, and it does not correlate well with either human judgement or model accuracy. SSIM (Wang et al., 2004) and MS-SSIM improve perceptual correlation; LPIPS (Zhang et al., 2018) improves it further using deep features.

For deployment work the honest metric is neither: measure your model's output change, not the pixel change. A shift of 3 dB matters or does not depending entirely on what the model does with it.

Formats worth knowing

Format	Loss	Notes
JPEG	lossy	8-bit, no alpha, universal support
PNG	lossless	8/16-bit, alpha, poor on photographs
WebP	both	lossy mode subsamples chroma like JPEG
AVIF	both	AV1 intra coding; far better rate-distortion, slower encode
JPEG XL	both	lossless JPEG transcode, ~20 percent smaller, patchy support
TIFF	usually lossless	16-bit and multi-page; the scientific-imaging default
DICOM	lossless typically	medical; 12–16 bit with windowing metadata that must be applied

The bit depth row matters more than it looks. Medical and satellite imagery is 12 or 16 bit. Converting to 8-bit JPEG for a pipeline discards dynamic range that the task depends on, and the windowing parameters used for that conversion become an undocumented part of your preprocessing.

EXIF orientation

Tag 274 (Orientation) takes values 1 through 8, encoding the eight combinations of rotation and mirroring. Value 1 is "as stored". Decoders differ: PIL.Image.open ignores it, browsers apply it, cv2.imread ignores it, and torchvision.io.read_image ignores it. Any pipeline mixing two decoders has an orientation bug waiting.

The corresponding failure in a labelling pipeline is worse: annotators using a browser see the rotated image and draw boxes in rotated coordinates, while the training loader reads the unrotated array. The result is a dataset with a subset of systematically wrong labels and no error anywhere.

References

ITU-T T.81 / ISO-IEC 10918-1, Digital compression and coding of continuous-tone still images, 1992
Wallace, The JPEG Still Picture Compression Standard, Communications of the ACM, 1991
Wang et al., Image Quality Assessment: From Error Visibility to Structural Similarity, IEEE TIP 2004
Dodge and Karam, Understanding How Image Quality Affects Deep Neural Networks, QoMEX 2016 — arxiv.org/abs/1604.04004
Hendrycks and Dietterich, Benchmarking Neural Network Robustness, ICLR 2019 — arxiv.org/abs/1903.12261
EXIF 2.32 specification, CIPA DC-008 — orientation tag semantics

What to learn next

Train and serve preprocessing mismatch — the other half of the input-fidelity problem.
Testing robustness to corruptions — measuring what compression costs your accuracy.
How images are stored — the pixel representation underneath all of this.

🟢 Beginner — No maths. Plain English.

The short answer#

The analogy you have lived#

Two kinds of format#

The part that surprises everyone#

What the damage looks like#

The other one: the sideways photo#

Where you have seen this#

The honest part#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

Measuring what compression costs#

The finding that should change how you save images#

What to actually do#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

What JPEG actually does#

Generation loss#

Effects on models#

PSNR, and why it is a weak instrument#

Formats worth knowing#

EXIF orientation#

References#

What to learn next#

Beginner — No maths. Plain English.

The short answer

The analogy you have lived

Two kinds of format

The part that surprises everyone

What the damage looks like

The other one: the sideways photo

Where you have seen this

The honest part

Remember this

What to learn next

Developer — Code and libraries.

Setup

Measuring what compression costs

The finding that should change how you save images

What to actually do

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

What JPEG actually does

Generation loss

Effects on models

PSNR, and why it is a weak instrument

Formats worth knowing

EXIF orientation

References

What to learn next