Power and thermal limits

Where the energy goes#

Dynamic power in CMOS is

$$ P_{\text{dyn}} = \alpha C V^2 f $$

• $\alpha$ — activity factor, the fraction of transistors switching per cycle.
• $C$ — total switched capacitance.
• $V$ — supply voltage.
• $f$ — clock frequency.

Static leakage $P_{\text{leak}} = V I_{\text{leak}}$ adds to this, and grows superlinearly with temperature — which is what makes thermal runaway a real design concern rather than a theoretical one.

The quadratic in $V$ is the whole basis of dynamic voltage and frequency scaling. Sustaining a higher $f$ requires a higher $V$, so power grows roughly as $f^3$ while work grows linearly in $f$. Energy per operation therefore rises with frequency, and the cheapest way to do a fixed amount of work is slowly.

This produces the race-to-idle versus crawl question. Race-to-idle wins when static power and fixed platform overhead dominate, because finishing early lets the whole system enter a deep sleep state. Crawling wins when dynamic power dominates and there is no deep state to reach. For inference on a phone, race-to-idle usually wins, because the display and memory subsystem impose a large fixed cost while anything is awake at all.

Thermal dynamics#

A first-order lumped model:

$$ C_{\text{th}} \frac{dT}{dt} = P(t) - \frac{T - T_{\text{amb}}}{R_{\text{th}}} $$

• $C_{\text{th}}$ — thermal capacitance in J/K, the heat the package absorbs per degree.
• $R_{\text{th}}$ — thermal resistance to ambient in K/W.
• $T_{\text{amb}}$ — ambient temperature.

The steady-state rise is $\Delta T = P R_{\text{th}}$, reached with time constant $\tau = R_{\text{th}} C_{\text{th}}$. For a phone $\tau$ is on the order of minutes; for a bare Raspberry Pi, tens of seconds; for a microcontroller, effectively instant.

Two consequences that decide benchmark design. A benchmark shorter than $\tau$ measures the transient and reports a number the device cannot hold. And ambient temperature enters linearly — the same phone in a 20 °C room and a 40 °C room has substantially different sustained throughput, which is not a minor consideration for deployments in India, the Gulf or an unventilated equipment cabinet.

Sustained performance is bounded by $P_{\max} = (T_{\text{limit}} - T_{\text{amb}}) / R_{\text{th}}$ regardless of what the silicon could do.

The energy hierarchy, again#

From Horowitz (2014), at 45 nm, the numbers that determine where optimisation effort belongs:

A DRAM access costs more than 170 fp32 multiplies. This single ratio explains the shape of every efficient accelerator: they are built around keeping data resident and reusing it, not around doing arithmetic faster.

It also reframes quantisation. The arithmetic saving from int8 over fp32 is roughly 30x per multiply, but the memory saving is 4x on a term that is three orders of magnitude larger. Quantisation is a memory-traffic optimisation that happens to also reduce arithmetic energy, and this is why it helps even on hardware with no integer speed-up.

The radio tail#

Balasubramanian et al. (2009) measured the energy cost of cellular transfers directly and found the dominant term to be the RRC state machine's tail, not the transmission. After a transfer, 3G held a high-power state for 5–12 seconds; LTE's tail is shorter but present, and 5G NR adds its own inactivity timers.

The energy for $n$ transfers separated by more than the tail is $n \cdot P_{\text{radio}} (t_{\text{tx}} + t_{\text{tail}})$, whereas $n$ coalesced transfers cost $P_{\text{radio}} (n\, t_{\text{tx}} + t_{\text{tail}})$. For small payloads the difference approaches a factor of $n$.

This is why "send it to the server" is an energy decision and not only a latency one, and it is the quantitative form of the argument made in why run a model on the device.

Reporting standards#

The literature is poor here. A checklist that makes an edge benchmark reproducible:

1. Sustained throughput over a period exceeding the thermal time constant, plus the burst figure separately.
2. Ambient temperature and cooling arrangement.
3. Power source: battery or mains, and the battery's state of charge.
4. Frequency governor and, where available, the throttling state before and after.
5. Joules per inference, with the measurement method stated — board-level, package-level or estimated.
6. Device model and firmware version, since throttling ladders change between firmware revisions.

MLPerf Tiny (Banbury et al., 2021) includes an energy measurement track with a specified harness, and it is the only widely used benchmark in this space that treats energy as a first-class result rather than a footnote.

Reading#

• Horowitz, Computing's Energy Problem, ISSCC 2014.
• Balasubramanian, Balasubramanian and Venkataramani, Energy consumption in mobile phones: a measurement study and implications for network applications, IMC 2009.
• Banbury et al., MLPerf Tiny Benchmark, NeurIPS Datasets and Benchmarks 2021 — arxiv.org/abs/2106.07597
• Chen, Emer and Sze, Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks, ISSCC 2016 — dataflow design driven entirely by the memory hierarchy above.
• Yang, Chen and Sze, Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning, CVPR 2017 — pruning by measured energy rather than by parameter count, and the two orderings differ.

What to learn next#

• Latency and throughput — the measurement discipline this lesson depends on.
• Model compression — the levers you reach for once you know what is binding.
• Monitoring and model drift — watching a deployed model after it leaves your hands.