Learn AI Create the Future — with Pranay
Main site

GPUs: Memory, Scheduling and Cost

Reading GPU utilisation honestly

The GPU utilisation percentage tells you whether the GPU is doing anything at all in a given moment. It does not tell you whether it is being used well.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

GPU utilisation tells you whether the GPU is doing anything at a given moment. It does not tell you how efficiently that work is being done.

The analogy you have already lived

Imagine checking whether a shared kitchen stove is "on" by glancing at it from across the room. You can tell if the flame is lit. You cannot tell from that glance whether the pot is nearly empty. Or whether the flame is far bigger than the pot needs. Or whether the cook uses every minute the stove is on well.

"The stove is on" and "the stove is being used well" are two completely different questions. GPU utilisation answers the first one. Many people mistake it for an answer to the second.

Why it exists

Every GPU-serving setup eventually asks: is this GPU actually earning its cost? The first number most people reach for is GPU utilisation. It is a percentage, shown by monitoring tools, that seems to answer this directly.

It answers a narrower question than it appears to: was some GPU work running during this moment, yes or no. A GPU showing 95% utilisation can still be doing far less useful work per second than it is capable of.

How it works

   every fraction of a second, the GPU is asked:

   "was ANY GPU instruction running just now?"

        yes  ------>  counts toward "busy" time
        no   ------>  counts toward "idle" time

   utilisation % = busy time / total time, over some window

Notice what this never asks: how much of the GPU's actual capacity that instruction used. A tiny piece of work that barely touches the GPU's power counts exactly the same as a huge piece of work using nearly all of it. Both count identically as "busy."

A real example you have seen

A phone's battery percentage tells you roughly how much charge is left. It does not tell you which app is draining it fastest. Nor whether that drain is reasonable for what the app is doing. You need a second, more detailed view to actually understand what is happening. Utilisation percentage has exactly the same limitation.

The honest part

A GPU sitting at 95% utilisation, serving a small model on tiny requests, can be almost entirely wasted. It is busy in the narrow sense the number measures, while doing a small fraction of the useful work the hardware is capable of. Treating one summary number as the whole picture is a common mistake in running GPU infrastructure. It is also an expensive one.

Remember this

  • GPU utilisation means "was something running," not "how well is the GPU being used."
  • A high percentage can still mean wasted capacity, not full value for money.
  • Treat it as a first check, not a final answer, about whether a GPU is earning its cost.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch

This uses the nvidia-smi command-line tool, which ships with the NVIDIA driver. If it is not on your machine's PATH, this script's subprocess calls will fail. That is expected on a machine with no NVIDIA GPU.

Watching utilisation while real work runs

watch_utilisation.py
import subprocess
import threading
import time
import torch

def read_utilisation():
    out = subprocess.run(
        ["nvidia-smi", "--query-gpu=utilization.gpu,memory.used", "--format=csv,noheader,nounits"],
        capture_output=True, text=True, timeout=5,
    )
    util, mem = out.stdout.strip().split(",")
    return int(util), int(mem)

before_util, before_mem = read_utilisation()
print(f"before: {before_util}% util, {before_mem} MiB used")

stop, readings = False, []

def sampler():
    while not stop:
        readings.append(read_utilisation())
        time.sleep(0.15)

t = threading.Thread(target=sampler)
t.start()

# Real, sustained GPU work: large matrix multiplications, back to back.
a = torch.randn(4096, 4096, device="cuda")
b = torch.randn(4096, 4096, device="cuda")
start = time.perf_counter()
while time.perf_counter() - start < 2.0:
    c = a @ b
torch.cuda.synchronize()

stop = True
t.join()
del a, b, c
torch.cuda.empty_cache()
time.sleep(0.5)  # let queued work drain before checking "idle" again

busy_util = [u for u, _ in readings]
after_util, after_mem = read_utilisation()
print(f"samples taken during the 2s of GPU work: {len(readings)}")
print(f"utilisation % during work: min={min(busy_util)} max={max(busy_util)} avg={sum(busy_util)/len(busy_util):.0f}")
print(f"after (work finished, cache cleared): {after_util}% util, {after_mem} MiB used")
Output
before: 4% util, 5312 MiB used
samples taken during the 2s of GPU work: 47
utilisation % during work: min=4 max=100 avg=96
after (work finished, cache cleared): 3% util, 5468 MiB used

A real run, on this machine, with a real NVIDIA GPU. The pattern is genuine, and will show on any real GPU: low before, high while continuous work runs, low again after. The exact numbers depend on the specific GPU, on this exact matrix size, and on whatever else happens to run on the machine at the same time.

Line-by-line walkthrough

nvidia-smi --query-gpu=utilization.gpu,memory.used. Asks the driver directly for its own utilisation figure, in a script-friendly comma-separated format. This is the same underlying number every GPU monitoring dashboard is built on top of.

A background thread sampling every 0.15 seconds, while the main thread runs continuous GPU work. Utilisation is meaningless as a single point-in-time snapshot for bursty workloads. Sampling repeatedly while real work runs is what makes it a trustworthy measurement, rather than a lucky guess.

memory.used reports everything on the GPU, not just this script. If other processes are using the same physical GPU, their memory use is included in this number too. That is a real fact about the metric, not a bug in this script.

Common mistakes

Treating one snapshot as the whole story. GPU work is often bursty — busy for a moment, idle the next. A single reading can land on either extreme by chance. Sample repeatedly, over a real time window, before drawing a conclusion.

Confusing "GPU utilisation" with "memory utilisation." They are two entirely separate numbers. A GPU can be at 100% compute utilisation while barely using its memory. Or the reverse: holding a huge model in memory, while doing almost no compute work most of the time.

Assuming low utilisation always means the GPU is wasted. A GPU serving requests only occasionally, with real idle gaps between them, can be entirely appropriate for genuinely low traffic. Low utilisation is a question to investigate, not automatically a problem to fix.

Not accounting for other processes sharing the GPU. On a shared machine, another program's activity shows up in your readings. Before concluding your own workload is inefficient, check what else is actually running on the same GPU.

Try it yourself

Change the matrix size from 4096 to 512 and rerun. Compare the average utilisation reading. A much smaller amount of work per instruction can still show a surprisingly high utilisation percentage — exactly the trap this lesson is about.

What to learn next

Researcher — Mathematics and papers.

What the utilisation counter actually measures

NVIDIA's driver reports utilization.gpu as the percentage of time, over the last sampling period, during which at least one kernel was executing on the device. It is fundamentally a time-occupancy metric, not a throughput or efficiency metric. It carries no information about:

  • Occupancy — the fraction of the GPU's streaming multiprocessors actually active during that kernel.
  • Arithmetic intensity realised — whether the kernel is compute-bound or memory-bandwidth-bound, and how close it runs to either theoretical peak.
  • Tensor Core utilisation specifically — a kernel can occupy the GPU's general compute path at 100% "utilisation" while never touching the specialised Tensor Cores that dominate modern peak FLOPs figures.

Better signals, in increasing order of detail

  • SM (streaming multiprocessor) occupancy, reported by nvidia-smi dmon or the NVIDIA Management Library (NVML) directly, measures the fraction of available parallel execution units actually active. It is a materially better proxy for "is this GPU doing a lot of work" than time-occupancy alone.
  • Achieved memory bandwidth versus peak. For memory-bound inference workloads (see the discussion of decode being memory-bandwidth-bound in choosing a GPU for inference), this is often the more meaningful ceiling to measure against — more than compute FLOPs at all.
  • Kernel-level profiling. NVIDIA Nsight Systems and Nsight Compute break down time spent per kernel, per memory transfer, and per synchronisation point. This is what real capacity-planning and optimisation work is actually done against, rather than the coarse driver-level percentage.
  • Application-level throughput. Ultimately, the metric that matters for a serving system is requests or tokens processed per second, at an acceptable latency. Utilisation percentage is only a weak, indirect proxy for that.

Why this matters for capacity planning specifically

Provisioning decisions made purely against utilisation percentage systematically over- or under-provision, because the same reported percentage can correspond to very different real throughput depending on kernel efficiency. A team observing "our GPU is at 90% utilisation, we need more GPUs" without checking occupancy or achieved bandwidth may end up paying to scale out an inefficient kernel. Fixing the kernel is a substantially cheaper option, when it is available. Profiling inference code covers the practical tooling for finding that distinction.

Reading

  • NVIDIA, Nsight Systems and Nsight Compute documentation — the standard tooling referenced above
  • NVIDIA Management Library (NVML) API reference — the interface nvidia-smi itself is built on, for programmatic access to the same and deeper metrics

What to learn next