Microduck StudioIndependent project

Microduck Studio books · Book 2 of 4 · Intermediate

How a Microduck Learns

How does a duck learn to walk when nobody writes the walk? Ten lessons, with the real numbers.

Who it is for: teens and adults, and anyone who finished Book 1. You never need to write code, though a few notes show commands for you or a friend who codes. Every number has a source, most of them Pollen's training code, and the notes say where.

  1. Observe, act, score, repeat
  2. 61 numbers
  3. 14 motors, 14 numbers
  4. A brain made of sums
  5. Points for what you want
  6. Practice, at scale
  7. Small steps uphill
  8. Train on many worlds
  9. From training to a robot
  10. Your first project: bounce a ball

How a lesson works: read the left page and commit to a prediction, then run the steps on the right one at a time. The notes under each lesson go further, into what to try in Microduck Studio, Kith or Pollen Robotics' training code and where every number comes from.

Lesson 1 · The loop

Observe, act, score, repeat

Every reinforcement learning problem has the same shape. An agent, here the duck's brain, gets an observation of the world and picks an action. The world moves on, and hands back the next observation and a reward: one number for how good that moment was.

The duck's loop runs 50 times a second. One turn of it is a step, and the steps from a reset until the duck falls, or time runs out, make an episode.

The agent is not trying to win one step but to collect as much reward as it can from the steps ahead, counting near ones more than far ones. With Pollen's settings about the next 2 seconds count most, which is why a duck learns to take a small loss now, like a careful step, for more later.

observation → action → next observation and reward → again

Predict

Suppose an episode lasts 20 seconds. How many decisions does the duck make in it?

Check your prediction

1,000: 50 decisions a second for 20 seconds, each one an observation in and an action out, and each earning a reward.

The pop-up

  1. Observe. Numbers flow from the duck's sensors into its brain.
  2. Act. The brain sends numbers out to the motors, and the duck moves.
  3. Score. The world hands back a reward for that moment, and the episode's total grows.
  4. Repeat. Round and round at 50 steps a second, until the duck falls and the episode resets.
The loop at its real rate. The reward per step is drawn for the lesson.

Big idea: learning to walk is learning to pick actions that pay off over the next few seconds, not just the next step.

Lesson 2 · The observation

61 numbers

The duck's brain never sees the room, only a list of 61 numbers, 50 times a second.

3 say how fast its body is turning (a gyroscope) and 3 which way is down. 14 say where each joint is, 14 how fast each is moving, and 14 repeat the actions the brain chose last step. The last 13 are its orders: how fast to walk and turn (3), where to point its head (4), and how to hold its body (6).

Every walking, standing and trick brain Pollen Robotics trains reads this same list in the same order, so the robot can hand over from one brain to another, say from walking to a kick.

Some things are missing on purpose: the robot has no direct way to measure its own speed, and the ball-kicking brain never sees the ball.

3 + 3 + 14 + 14 + 14 + 13 = 61

Predict

Why might the list include the duck's own last actions?

Check your prediction

Its joints lag behind. A joint takes time to reach its target, so the joint readings alone are always a little behind, but its last orders say where the joints are heading. They fill in the rest, and they also let the brain keep its moves smooth.

The pop-up

  1. Turning. The first 3 bars light up: the gyroscope's spin, around three axes.
  2. Down. The next 3 show which way gravity points as the body tilts.
  3. Joints. 14 bars of joint angles, then 14 of joint speeds, move with the legs.
  4. Memory. 14 bars echo the actions the brain sent last step.
  5. Orders. The last 13: walk and turn speeds, head pose, and the body pose, mostly zeros.
The real layout of the 61 numbers, as a bar code beside the duck. The bar heights are drawn for the lesson.

Big idea: a robot knows only what its observation tells it, so choosing those numbers is half of designing the robot's brain.

Lesson 3 · The action

14 motors, 14 numbers

Each step, the brain answers with 14 numbers, one per motor, always in the same order: the left leg's 5 joints, then 4 in the neck and head, then the right leg's 5.

A number is not a speed or a push. It is a target, where the joint should go, measured from a home pose. The motor's own small controller then drives the joint toward it, the way a thermostat heats a room toward the temperature you set.

target = home pose + action

Predict

The brain gives the left knee a new target. When does the knee get there?

Check your prediction

A little later, and not quite exactly. Real motors are not instant: they lag behind their orders, their gears have a little slack, and they get weaker as the battery runs down and as they spin faster. The simulator models the lag and the weakening, and every main task has a twin that adds the gear slack, because at this size the motor is most of the difference between pretend and real. A brain trained on an instant, perfect motor would fall on the real one.

The pop-up

  1. The order. The motors light up in the brain's order: left leg, neck and head, right leg.
  2. A target. The brain asks for a new leg pose. A ghost leg shows where it should go.
  3. Follow. The real leg follows the ghost, behind it.
  4. Arrive. Even when it arrives, the joint can rock a little in its gears.
The real joint order. The leg's moves are exaggerated so you can see them.

Big idea: a robot's actions are wishes to its motors, so training has to happen with motors as slow and sloppy as the real ones.

Lesson 4 · The policy

A brain made of sums

The duck's brain is a policy: a function that turns 61 numbers into 14. Pollen's is a small neural network: 61 inputs, three layers of 512, 256 and 128 units, then 14 outputs.

Each unit multiplies its inputs by weights, adds them up with a bias, and bends the total with a simple curve. That is all a unit does.

Before the numbers go in, each is shifted and scaled by the average and spread it had in training, so a joint angle and a spin speed arrive on a similar scale. That rescaling is part of the brain, so it is built into the file the robot runs.

61 → 512 → 256 → 128 → 14

Predict

Training changes the brain. What exactly changes?

Check your prediction

The weights and biases. About 198,000 of them, set by training, are the difference between a duck that falls and one that walks, while the shape is chosen before training and never changes. Training turns the dials, and keeps the rescaling's averages up to date.

The pop-up

  1. Inputs. 61 numbers arrive, rescaled.
  2. Layer one. Each of 512 units adds up all 61 inputs, weighted. One dot here stands for 8 units.
  3. Deeper. The signal passes through 256 units, then 128.
  4. Outputs. 14 numbers come out, and the duck's joints follow them.
  5. Turn the dials. Drag the slider to scale every weight, and watch the outputs and the pose change.
The shape is Pollen's network; the weights in this picture are random.

Big idea: a policy is just arithmetic, and training is the search for the right numbers.

Lesson 5 · The reward

Points for what you want

Nobody writes the walk. They write the score, and reinforcement learning (RL) finds a walk that earns it.

The walking duck's score is a sum of terms, worked out every step. It earns points for matching the speed it was asked for, keeping its body upright, lifting its feet and pointing its head where asked. It loses points for jerky actions, sliding its feet, and its legs knocking into its body.

Get the score wrong and the duck finds the loophole, the way a student can learn the test instead of the subject.

"RL optimizes the letter of the reward."

Predict

The duck loses points for every tiny head wobble, and walking always makes its head wobble. What does training find?

Check your prediction

It stands still. When its trainers charged for every wobble of its heavy head, the best score came from not walking at all. The wobble could not be escaped while walking, so stopping was the cheapest way to avoid the charge. The fix charges only for the steady droop the duck can correct, so a good walk pays nothing for it.

The pop-up

  1. Earn. Bars rise for the terms that pay: speed, turning, upright, feet, head.
  2. Pay. Bars drop for the costs: jerky actions, sliding feet, legs knocking the body.
  3. Add up. The step's score is the sum, and training wants it high over the episode.
  4. Too strict. The head charge tightens. The duck slows, its feet stop leaving the ground, and it stands.
  5. The fix. Charge only the steady droop, which a good walk does not pay. The duck can walk again.
The walking task's main terms, with how much each term counts; the full score has a few more, such as charges for feet that skim the floor. The bar heights move for the lesson.

Big idea: a reward is a wish written in arithmetic, and the robot will grant exactly what it says.

Lesson 6 · Training

Practice, at scale

Training runs 4,096 pretend ducks at once on a graphics card. Each duck takes 24 steps, then the brain learns from all of them together, and that is one iteration: 98,304 steps of experience.

A usable walk takes about 1 to 2 hours. Pollen's guide says simple tricks need about 1,000 iterations, while walking and getting up need 4,000 to 6,000. A copy of the brain is saved every 250 iterations.

Trainers watch the score climb, and they watch the duck too, because the score can climb while the trick never happens.

4,096 ducks × 24 steps = 98,304 steps per iteration

Predict

One iteration is 24 steps for each duck. At 50 steps a second, how much of each duck's life is that?

Check your prediction

About half a second: 24 steps at 50 a second is 0.48 seconds of each duck's life, but short bursts from thousands of ducks give the brain a wide view of what it is doing.

The pop-up

  1. Collect. 4,096 ducks each take 24 steps. The bar fills with experience.
  2. Learn. The brain updates from all of it at once.
  3. Repeat. Iteration after iteration, the score on the chart climbs.
  4. Save. Every 250 iterations, a copy of the brain is kept.
  5. Watch it. One duck plays the latest copy, because a score is not a walk.
The counts are Pollen's settings. The chart's curve is drawn for the lesson, not taken from a run.

Big idea: thousands of short, parallel tries are what make learning by trying affordable.

Lesson 7 · The update

Small steps uphill

While it practises, the brain tries numbers around its average, so every action has a chance. After each round of practice it is nudged so that actions that went better than expected become more likely, and actions that went worse become less likely.

The method is PPO, proximal policy optimization, and proximal means near: it will not move the policy far in one go. Once a change would make an action more than 20% more or less likely than before, the update stops pushing it.

Past a 20% change in an action's likelihood, an update gains nothing more.

Predict

Why does PPO take small steps?

Check your prediction

Its practice came from the old brain. Change the brain too much and that practice no longer describes it, and learning falls apart. Small steps keep the new brain close enough to the old one that its practice still counts. Many small steps climb further than one wild leap.

The pop-up

  1. Practice. The brain, a marble on a hill of scores, tries actions around where it stands.
  2. Leap. An unclipped update leaps too far, past the top and off the edge.
  3. Clip. Back to the start. Now each step stays inside the safe band.
  4. Climb. Step by small step, the marble reaches the top.
Drag the slider to widen or narrow the band. The 20% band is the walking task's; the hill is a picture.

Big idea: learning from your own practice works best in small, careful steps.

Lesson 8 · Sim to real

Train on many worlds

A brain trained on one perfect world learns that world, flaws and all, so every pretend duck gets a slightly different one. Some differences are drawn once for each duck and some again each time it starts over, and the idea is called domain randomization.

The floor's grip ranges from 0.7 to 1.3 times normal. The main body's weight changes by 5% either way, and its balance point shifts by up to 15 mm. Every 3 to 6 seconds the duck is shoved, by up to 0.3 m/s. Its tilt sensor is mounted wrong by up to 6°, and each joint reading is off by a small, fixed amount.

A brain that walks in all of these treats the real robot as one more variation.

Train on a spread of worlds, so the real one falls inside it.

Predict

The real duck's tilt sensor sits 5° off, always the same way. Does the randomization take care of that?

Check your prediction

No: it has to be corrected on the robot. Each pretend duck's sensor is tipped a fixed amount in a random direction, so the brain learns to cope with an error of that size, not with one steady tilt that every duck shares. Pollen's training settings leave the real board's steady offset to be corrected on the robot, before the brain sees the numbers.

The pop-up

  1. One world. Nine ducks stand on the same perfect floor.
  2. Grip. Each floor changes, from slippery to grippy.
  3. Body. Each duck gets a little heavier or lighter, with its balance point moved.
  4. Shoves. Every few seconds, a shove from a random side.
  5. Senses. Each duck's tilt sensor is tipped a different way.
The ranges are the walking task's own, and the tilts are drawn three times larger so you can see them. Which duck gets which is drawn at random.

Big idea: robustness is trained, by refusing to let the brain get used to any one world.

Lesson 9 · Deployment

From training to a robot

A trained brain leaves the simulator as a file, and every step from there is a check.

Export writes an ONNX file, a format many programs can run. Rehearse drives that file in a separate simulator on an ordinary computer, with the same motor model. Publish checks the file takes 61 numbers and gives 14, refuses one that outputs nonsense or a constant, and uploads it with a manifest that says how to run it.

On a robot, one command installs it and another runs it, and on a phone, Microduck Studio opens the same file and shows you what is inside.

train → export → rehearse → publish → install

Predict

Why must the file be exported with the project's own script, never converted by hand?

Check your prediction

The rescaling. Export writes the file with the input rescaling built in. Training's own playback applies it anyway, so the brain looks fine there. A file without it fails wherever it runs alone, on the robot or in the rehearsal script, because the brain sees numbers of the wrong size. That is why there is exactly one safe path.

The pop-up

  1. Train. The brain comes off the graphics card as a checkpoint.
  2. Export. It becomes an ONNX file, with its rescaling built in.
  3. Rehearse. A laptop drives it in physics, with the motor model.
  4. Publish. 61 in, 14 out, no nonsense: checked, then uploaded with a manifest.
  5. Install. One command puts a walk on a robot and drives it; a trick takes a second command to run.
The real path, station by station. Each stamp is a check the tools really make.

Big idea: a brain is only as good as the path that carries it to the robot.

Lesson 10 · Your project

Your first project: bounce a ball

A good first project changes as little as possible. Here is one to try: teach the duck to bounce a small ball off its beak.

Start from what exists. Pollen's kicking task already puts a 70 mm, 15 g ball in front of the duck, and the ground-pick task already bends it to touch the floor with its beak.

Change one thing at a time. The kicker cannot see the ball, but bouncing needs to know where it is, so the brain needs new inputs.

Then expect loopholes: Pollen's guide expects 2 to 5 rounds of fixing them.

Copy the closest task. Change one thing. Watch what it does.

Predict

You reward the duck for every step the ball is high above it. What might it learn?

Check your prediction

To hold it still. Balancing the ball on its head keeps it high every step, which pays more than any bounce. So reward each upward bounce off the beak, not the ball being high: pay for the bounce itself, the moment the beak sends the ball up.

The pop-up

  1. Start here. Two tasks that exist: kick a ball, and touch the ground with the beak.
  2. Add sight. The ball's position joins the inputs, and the list grows past 61.
  3. Pay for bounces. A reward each time the beak sends the ball up.
  4. Loophole. Reward the ball being high instead, and the duck balances it on its head.
  5. Fixed. Back to paying for bounces, and the ball bounces.
A plan, not a result: nobody has trained this yet. The ball is the kicking task's.

Big idea: start from a working task, change one thing, and read what the duck does, not just its score.

Before this: Book 1, Meet the Microduck. Next: Book 3, Training Microducks, for the equations, the reward rules, the motor model and honest statistics. All four books are on the shelf.