Robotics, Control and Autonomy
Sim-to-real transfer
A controller or policy trained in simulation often stops working the moment it meets a real robot, because simulation never modelled the world with full accuracy.
- 10 min read
- 3 reading levels
- Published
Read these first
On this page 6
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Sim-to-real transfer is getting a robot skill learned in simulation to keep working on a real robot.
Think about learning to ride a bicycle on an indoor training stand. Wheels spinning in place, no wind, no traffic, no need to balance sideways at all. You get comfortable pedaling and steering. Then you take the real bicycle onto an actual road. Your same muscle memory suddenly has to fight balance, bumps, and wind — none of which the stand ever taught you.
Robots trained purely in simulation hit the same wall. This gap between the training world and the real one has a name: the reality gap.
Why it exists
Training a robot skill directly on real hardware is slow, and can break the robot, or hurt someone standing nearby. Every real-world trial takes real seconds and carries real risk. Simulation fixes both: a physics engine can run thousands of trials in parallel, faster than real time, with zero physical risk.
The catch is that simulators are approximations. They get friction slightly wrong. They get how motors respond to commands slightly wrong. Camera images look almost, but not exactly, like real ones. A skill perfected against these small wrong assumptions can fail the moment it meets the real, unapproximated version.
How it works
SIMULATION REALITY GAP
(millions of cheap, safe trials) --> trained skill --> performs worse
| than in training
|
DOMAIN RANDOMIZATION: vary friction, mass, lighting,
sensor noise, delays -- during training, not only once
|
v
a skill that has seen many "realities," not only one --> usually
transfers betterThe core trick, domain randomization, deliberately makes the simulator less accurate on purpose. It trades narrow precision about one guess for rough accuracy across a wide range. A skill that has learned to cope with friction anywhere from very low to very high stops depending on friction being exactly right.
Where you have already seen it
- Warehouse picking robots are trained largely in simulation before ever touching a real shelf. A real trial-and-error phase on live inventory would be too slow, and too costly.
- Research demonstrations of a robot hand solving a Rubik's cube, or a four-legged robot walking over rubble, are trained almost entirely in randomized simulation first.
- Self-driving car companies log millions of simulated miles for every real mile driven, precisely because real trial and error carries real risk.
An honest note
The reality gap is not a solved problem. A policy that looks flawless in simulation can behave in a genuinely dangerous way on real hardware. It can hesitate, overcorrect, or move unpredictably in situations the simulator never modelled. No lesson-sized example can replicate the safety validation a real robotics team performs. Deploying anything that moves near people needs staged testing, and human oversight during early real-world trials. For vehicles and many industrial robots, it needs regulatory approval too. None of that is optional, and none of it is shown here.
Remember this
- Simulation makes training fast and safe, but it is never a perfect copy of reality.
- Domain randomization trains across many varied simulated conditions, not one precise guess.
- A skill that transfers still needs real-world testing before it is trusted, not instead of it.
What to learn next
- Imitation learning and behaviour cloning — a different route to a working skill, learned from real demonstrations instead of simulated trial and error.
- What is reinforcement learning? — the training method most sim-to-real work is built on.
- PID control, and when to use it instead — the controller in this lesson's example, and its own tuning limits.
Developer — Code and libraries.
Setup
No installs needed — the example is plain Python.
Minimal runnable code
A robot arm joint with real static friction — the resistance that has to be overcome before anything even starts moving. The simulator used to tune the first controller never modelled friction at all.
import numpy as np
def run(kp, ki, kd, friction, steps=1000, dt=0.02, target=1.0, mass=1.0):
pos, vel, integral = 0.0, 0.0, 0.0
prev_error = target - pos
for _ in range(steps):
error = target - pos
integral += error * dt
derivative = (error - prev_error) / dt
drive_force = kp * error + ki * integral + kd * derivative
# Static friction: below a threshold force, the arm does not move at all.
if abs(vel) > 1e-4:
net_force = drive_force - np.sign(vel) * friction
elif abs(drive_force) > friction:
net_force = drive_force - np.sign(drive_force) * friction
else:
net_force, vel = 0.0, 0.0
vel += (net_force / mass) * dt
pos += vel * dt
prev_error = error
return abs(target - pos)
REAL_FRICTION = 1.4 # a real joint's static friction, absent from the simple simulator
# Tuned in a FRICTIONLESS simulator -- pure PD looked perfect there.
sim_only = run(kp=36.0, ki=0.0, kd=12.0, friction=REAL_FRICTION)
# Tuned across a RANDOMIZED range of simulated friction (0 to 2.0), which
# exposes the need for integral action to overcome an unmodelled resistance.
randomized = run(kp=36.0, ki=8.0, kd=12.0, friction=REAL_FRICTION)
print(f"tuned in a frictionless simulator only -> final error on real arm: {sim_only:.4f} m")
print(f"tuned across randomized simulated friction -> final error on real arm: {randomized:.4f} m")tuned in a frictionless simulator only -> final error on real arm: 0.0389 m tuned across randomized simulated friction -> final error on real arm: 0.0012 m
What actually happened
The frictionless-simulator controller gets permanently stuck 3.9cm short of target. Its drive force at that point is too small to break static friction. Running it for any longer changes nothing — the sim never taught it that this kind of resistance could exist. The randomized controller carries integral action strong enough to eventually build up the extra force needed to break through. Training saw friction values large enough to demand it.
- This is a classical-control stand-in for the real idea. Actual sim-to-real robotics work trains full neural network policies across thousands of randomized physics simulations running in parallel, not one hand-derived formula.
ki=8.0here was picked by hand for the demonstration. In real domain randomization, the training process itself discovers what a policy needs, across the randomized range — nobody hand-picks it.
Common mistakes
Randomizing too little. If the training range for friction had been 0.0 to 0.05, it would never come close to representing REAL_FRICTION = 1.4. The "randomized" controller would then fail exactly like the naive one.
Randomizing too much. Extremely wide ranges can make training find an overly conservative policy — safe everywhere, excellent nowhere. Real teams treat the randomization range as a tuned hyperparameter, checked against calibration data from the actual robot when it exists.
Treating sim-to-real success as the finish line. A policy that performs well on one real robot, once, is not evidence it is safe across many robots, many days, and unexpected conditions. Repeated, monitored real-world testing comes after transfer, not instead of it.
Try it yourself
Change REAL_FRICTION to 3.0 — a stickier, more worn real joint than either controller was tuned for. Compare how much both errors grow. The randomized controller, tuned across 0 to 2.0, is being asked to extrapolate outside what it ever saw. That is a genuine limit of the technique, not only a limit of this toy example.
What to learn next
- Imitation learning and behaviour cloning — learning directly from real demonstrations instead of a simulated model.
- What is reinforcement learning? — how policies transferred by this technique are usually trained.
- Grasping and manipulation — a task where the reality gap is especially stubborn, because contact physics is hard to simulate well.
Researcher — Mathematics and papers.
Formalising the gap
Frame the problem as a Markov Decision Process (MDP) with an unknown true transition function T_real. Training happens against a simulator T_sim, an approximation. A policy pi optimal for T_sim is not, in general, optimal or even safe for T_real. The reality gap is formally the performance difference:
Gap(pi) = J_real(pi) - J_sim(pi)where J is expected return under each respective dynamics model. Nothing about optimizing J_sim bounds Gap(pi) without further assumptions.
Domain randomization
Tobin et al. (2017) introduced randomizing rendering parameters — textures, lighting, camera pose — for vision-based sim-to-real transfer with zero real training images. Peng et al. (2018) extended this to dynamics parameters — mass, friction, motor gain, latency — for a robot arm pushing task. Formally, instead of optimizing for a single T_sim, training optimizes expected return over a distribution of simulators:
pi* = argmax_pi E_{xi ~ p(xi)} [ J_{T_sim(xi)}(pi) ]where xi is a vector of randomized physics and rendering parameters, and p(xi) is a chosen distribution over plausible values. This directly discourages the policy from overfitting to any single simulator's specific quirks.
Automatic domain randomization and system identification
Manually choosing p(xi)'s range is itself brittle. OpenAI et al. (2019), in the Rubik's cube result, used Automatic Domain Randomization: expand the randomization range only as the policy demonstrates competence at the current range, letting difficulty scale with capability. A separate line of work — system identification — instead narrows the gap by estimating real physical parameters from real data and matching the simulator to them, trading generality for accuracy on one specific robot.
Where the gap is worst
Contact-rich tasks — grasping, in-hand manipulation, legged locomotion on uneven terrain — remain the hardest transfer cases. Contact dynamics — exactly when and how hard two surfaces touch — are numerically stiff and expensive to simulate accurately. Most physics engines use simplified contact models instead, which diverge from reality precisely where the task's success depends on getting contact right.
Cost
Massively parallel GPU-based simulators (Isaac Gym, MuJoCo XLA, Brax) run tens of thousands of simulated robot instances concurrently, reducing wall-clock training time for randomized policies from weeks to hours on a single machine. This parallelism is a major reason domain randomization became practical at the scale current research uses.
Key references
- Tobin, J. et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS.
- Peng, X. B. et al. (2018). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. ICRA.
- OpenAI et al. (2019). Solving Rubik's Cube with a Robot Hand. — arxiv.org/abs/1910.07113
- Zhao, W., Queralta, J. P., Westerlund, T. (2020). Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey. IEEE SSCI.
Current state
Sim-to-real remains an active research area, not a solved one. No general method guarantees a bounded reality gap for an arbitrary task, and results are typically reported per-task, per-robot, with real-world validation on limited hardware. Any claim of a universally reliable transfer method should be read with that limit in mind.
What to learn next
- Imitation learning and behaviour cloning — an alternative that sidesteps simulation by learning from real demonstrations directly.
- Vision-language-action models — large models increasingly trained on mixed real and simulated data, for exactly this reason.
- What is reinforcement learning? — the optimization framework underneath every method in this lesson.