Learn AI Create the Future — with Pranay
Main site

Robotics, Control and Autonomy

Safety in physical AI systems

A safe robot never depends on its AI being correct — it has independent, simple, hardware-level limits that hold even when the AI is wrong.

Read these first

On this page 6
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. The honest heart of this lesson
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A genuinely safe robot never assumes its AI will always be right — it has independent, simple limits that hold even when the AI is wrong.

Every elevator has a mechanical brake that can stop it even if every electronic system inside it fails. You have ridden in one without ever thinking about it. Nobody engineered that elevator around the assumption that the electronics would always work perfectly. They built a second, independent way to stop it, that does not depend on the first one at all.

Physical AI systems — robots, drones, self-driving cars — are built the same way, or they should be.

Why it exists

An AI model, however well trained, is a statistical system. It will sometimes be wrong, on some input, in some way nobody predicted. That is true of a spam filter too — but a spam filter's mistake means one email in the wrong folder. A robot's mistake can mean real physical harm.

The engineering answer is not "make the AI good enough to trust completely." No AI system has ever cleared that bar. Treating one as if it had is exactly how physical AI systems cause harm. The answer is defense in depth: build independent layers, so that one layer failing does not mean the whole system fails.

How it works

AI decision layer     -->  predicts, plans, decides what the robot should do
        |
        v
INDEPENDENT SAFETY LAYER   -->  simple, separately verified, cannot be
(hardware force limits,        talked out of its limit by anything
 geofences, emergency stops)    the AI layer requests
        |
        v
   physical actuator   -->  only ever moves within the safety layer's
                             bounds, no matter what was requested

The safety layer is deliberately simple. A hardware force limiter, or a mechanical stop that a motor physically cannot push past, is a physical fact, not a prediction. It is far easier to test and trust completely than the AI model is. It does not even need to understand the task to do its job.

Where you have already seen it

  • Collaborative robots ("cobots") on factory floors have hardware force and speed limits. Contact with a person cannot exceed an injury threshold, and that limit is enforced in hardware, not only in software.
  • Drones commonly have GPS geofencing that refuses to fly into restricted airspace, regardless of what a flight-planning algorithm requests.
  • Cars have physical, mechanical braking systems, entirely separate from any software driving assistance layered on top.

The honest heart of this lesson

Nothing in this platform's robotics lessons is safe to run on a real machine near a person — including the small demonstrations in each one. Every one of them was a simplified illustration of an idea, using invented numbers, on a laptop, with nothing physical attached. Real deployment of any of these ideas near people, on public roads, or in a home requires domain-expert engineering review and extensive real-world testing. For vehicles, medical devices, and most industrial robots, it also requires formal regulatory approval under established safety standards. That process exists precisely because a confident-looking AI system is not the same thing as a validated, safe one. A system that has never been tested against its failure modes has not earned anyone's trust yet.

Remember this

  • A safe robot does not rely on its AI being perfect; it has independent, simpler safety layers underneath it.
  • The safety layer should be simple enough to verify completely, even when the AI above it cannot be.
  • Real deployment near people needs testing, domain-expert validation, and often regulatory approval — none of which a lesson like this one can substitute for.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy

Minimal runnable code

A stand-in for a learned controller's output — mostly reasonable, with an occasional unusually large command. That kind of spike is a genuine, documented failure mode of learned systems, on inputs unlike their training data. An independent hardware safety layer catches it.

safety_layer.py
import numpy as np

rng = np.random.default_rng(3)

MAX_SAFE_FORCE = 40.0  # Newtons -- a hardware limit set by mechanical engineers,
                        # well below what could injure someone standing nearby

def learned_policy_command(n):
    normal = rng.normal(loc=15.0, scale=8.0, size=n)
    spike_chance = rng.random(n) < 0.03
    spikes = rng.uniform(60.0, 150.0, size=n)
    return np.where(spike_chance, spikes, normal)

def hardware_safety_layer(commanded_force):
    # Independent of the AI entirely -- cannot be talked out of its limit.
    return np.clip(commanded_force, 0.0, MAX_SAFE_FORCE)

N = 2000
raw_commands = learned_policy_command(N)
safe_commands = hardware_safety_layer(raw_commands)

dangerous_requests = np.sum(raw_commands > MAX_SAFE_FORCE)
still_dangerous = np.sum(safe_commands > MAX_SAFE_FORCE)

print(f"policy requested force above the safe limit: {dangerous_requests} times out of {N}")
print(f"force that actually reached the motor above the limit: {still_dangerous} times out of {N}")
print(f"largest force the policy ever asked for: {raw_commands.max():.1f} N")
print(f"largest force that ever reached the motor: {safe_commands.max():.1f} N")
Output
policy requested force above the safe limit: 66 times out of 2000
force that actually reached the motor above the limit: 0 times out of 2000
largest force the policy ever asked for: 148.6 N
largest force that ever reached the motor: 40.0 N

What actually happened

The simulated policy asked for a dangerously large force 66 times out of 2000, including one request of nearly 150 Newtons. Every single one of those requests was caught. Not because the policy got better, and not because anyone predicted exactly when it would misbehave — because the safety layer never trusted it in the first place.

  • np.clip() is doing the entire job here, and that is the point. A safety layer this simple can be reviewed, tested, and verified completely, in a way a full neural network policy cannot.
  • The 3% spike rate is invented for this example. Real teams estimate a controller's actual failure rate from extensive logged data, not a guess, and design safety margins with real uncertainty in that estimate.

Common mistakes

Putting the safety limit inside the same model that makes decisions. If the AI both proposes an action and checks whether it is safe, a single failure mode can defeat both jobs at once. Independence is the entire value of a safety layer.

Setting the safety limit from the AI's typical behaviour, instead of from what is actually safe. MAX_SAFE_FORCE should come from human injury thresholds and mechanical engineering, not from "the policy rarely goes above this anyway."

Treating a passed test suite as proof of safety. This example ran 2000 samples from a known distribution. Real deployed systems face inputs no test suite anticipated. This is exactly why the safety layer must not depend on the AI having been tested against every possible situation.

Try it yourself

Raise the spike rate from 0.03 to 0.15, simulating a much less reliable policy. Rerun, and check that the number of dangerous requests to the motor stays at zero regardless. A well-designed safety layer's job does not get harder as the AI above it gets worse — that separation is the entire design goal.

What to learn next

Researcher — Mathematics and papers.

Functional safety frameworks

Physical AI safety engineering draws on decades of functional safety practice from industries with earlier automation histories, rather than inventing new principles from scratch:

  • ISO 26262 (road vehicles) defines Automotive Safety Integrity Levels (ASIL A through D), requiring evidence proportional to a hazard's severity, exposure, and controllability.
  • ISO 10218 and ISO/TS 15066 (industrial and collaborative robots) specify force and pressure limits for human-robot contact, validated against biomechanical injury thresholds, enforced through hardware-verified limits, not solely software.
  • UL 4600 covers autonomous products generally, not only vehicles. It requires an explicit, auditable "safety case" — a structured argument, with supporting evidence, for why a system is acceptably safe for its intended use, rather than a fixed checklist.

A common thread across all three: safety is argued and evidenced, not assumed from good intentions or from a model's benchmark accuracy.

Fail-safe vs fail-operational design

A fail-safe system, on detecting a fault, moves to a known-safe state — stop, hold position, disengage. A fail-operational system must continue operating safely even after a fault, because stopping itself is not safe in context — an aircraft mid-flight cannot halt outright. Fail-operational design requires redundancy. Independent, ideally diverse implementations of safety-critical functions — different hardware, different software, sometimes different design teams — mean a single fault, or a single systematic design error, does not take down every layer at once.

The specification problem for learned systems

Traditional functional safety analysis (FMEA — Failure Mode and Effects Analysis; fault trees) assumes a system whose behaviour can be enumerated or bounded by design. Learned components resist this: a neural network's behaviour on an input outside its training distribution is not fully characterisable in advance by inspection. Current practice compensates by:

  • Treating the learned component as an untrusted proposer, with an independent, formally verifiable safety monitor as the final gate — the pattern the developer example above illustrates in miniature.
  • Runtime monitoring for out-of-distribution inputs, triggering a fallback to a simpler, verified controller when detected. Amodei et al. (2016), Concrete Problems in AI Safety, frame this general pattern as "safe exploration" and "robustness to distributional shift."
  • Formal verification of the safety monitor itself, where the monitor's logic is simple enough for exhaustive proof — the same reason the developer example's safety layer is a single clip operation, not a learned function.

Open problems

No consensus method exists for providing a quantified, statistically rigorous safety guarantee for a learned policy's behaviour across an open-ended real-world input distribution. Current best practice bounds risk through a combination of measures, not any single guarantee: restricted operational design domains (limiting where and when a system operates autonomously), independent hardware safety layers, extensive scenario-based and real-world testing, and staged deployment with escalating autonomy.

Key references

  • Amodei, D. et al. (2016). Concrete Problems in AI Safety. — arxiv.org/abs/1606.06565
  • Koopman, P. (2022). How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety.
  • ISO/TS 15066:2016. Robots and robotic devices — Collaborative robots.
  • UL 4600. Standard for Safety for the Evaluation of Autonomous Products.
  • Leveson, N. (2011). Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press — the broader systems-safety framework much of this field draws on.

What to learn next