Sim-to-Real Deep Reinforcement Learning

The problem

The hand-tuned MPC controller was reliable on flat ground and fell over on anything it had not been modelled for. Tuning it for each new surface did not scale.

What I built

A locomotion policy trained with PPO in Isaac Sim under domain randomization over mass, friction, and actuator delay, then deployed to hardware at 1 kHz alongside the existing safety layer.

How it works

• PPO in Isaac Sim with thousands of parallel environments; MuJoCo used for quick reward iteration.
• Domain randomization over payload mass, surface friction, and actuator latency — the last of these mattered most for transfer.
• Observation space restricted to what the real robot can actually measure, so nothing depends on privileged simulator state.
• Policy exported and run in the real-time control loop, with the existing safety controller retained as a fallback.

How it was tested

Policies are gated on a fixed suite of held-out simulated terrains before any hardware run, then evaluated on hardware against the MPC baseline on the same course.

Results

• Survived unmodelled terrain where the MPC baseline fell.
• Ran at 1 kHz on the target hardware.
• Recovered from pushes up to X N without falling.

What I learned

• Actuator-delay randomization was the single change that made transfer work. Mass and friction randomization alone produced a policy that looked great in sim and stumbled immediately on hardware.
• My first observation space accidentally included a simulator-only quantity. It trained beautifully and transferred not at all — a good lesson in auditing observations before rewards.