Duck Cards: the evidence

Summary

• Single cards mostly pass. Under randomized physics and sensing, 95 to 98% of the pre-registered short rows end within 10 cm and 15° of where they should.
• The pre-registered bar is not met. At Studio's 10 Hz state rate, heading sits on the 95% line (96.9% on one machine, 94.9% on another, 93.8% on a repeat), Walk back misses position (92.8% and 93.1%), and the stressed condition fails both.
• Long rows fail. Of the packs' rows of 10 cards or more, only 33 to 42% land under randomization and none under stress.
• Almost no falls. 0 falls in 4,128 rows on a quiet machine, and 2 in 4,800 when the same loop ran again as a control. Falls seen on an overloaded machine were the machine.
• The first fix tried was rejected. Calibrating heading drift while the duck stands still helped the stressed rows but added falls and broke rows with no drift at all (below).
• The second fix, the mat as a compass, is not adopted. It lifted long rows under randomization from 37% to 93%, but a handful of falls fail the rule, and its repeat missed its own validity gate by one row (below).
• The shipped loop, run twice, disagrees with itself. Seven verdicts differ between two runs of the same code, because rows in one simulated session share a duck. Later candidates are judged with a test that counts sessions, not rows (below).
• The third fix, merging cards, is kept as the base, with a small gain. Long rows under randomization at 10 Hz rose from 37% to 42%, and nothing got worse. Counted by session, the rise is not clear of noise (below).
• The fourth, a nest and chapters, is the new base. Long challenges play in chapters of up to 7 cards, and the child re-seats the duck on a nest square between them. Long rows under randomization at 10 Hz rose from 42% to 63%, worse nowhere counted by session (below). The stressed condition still fails.
• The fifth, the mat on the nest, is the new base in the sim. The compass from the second idea comes back, on a mat laid under each nest, and the mat now also measures the camera's own tilt. Long rows under randomization at 10 Hz rose from 63% to 97.5%, worse nowhere counted by session, and every prediction made in advance held (below). It is not in the app yet: it needs the duck's camera in the app and a printed mat.
• So closed-loop cards stay off on real ducks. In the shipped app a duck runs Walk and Turn as timed cards; the board rule is kept by the child and the paper board.

The question

Duck Cards are printed coding cards: a child lays Walk, Turn and Repeat cards in a row and the duck acts them out on a 30 cm grid. A timed card (walk for two seconds) does not land on a square: walking speed varies with the floor, the policy and the battery. A closed-loop card walks until odometry says one square and turns until the heading says a quarter turn. The question is whether that is good enough, measured, before any child relies on it.

System under test

• Simulator: Pollen Robotics' duck-sim with Pollen's own robotd answering JSON-RPC robot.move, and Pollen's v8 velocity-standing policy walking the duck. Odometry is robotd's own estimate; errors are judged against MuJoCo's true pose.
• Envelope: commands are clamped to Pollen's gamepad defaults, vx within 0.4 m/s, vy within 0.3 m/s, vyaw within 1.0 rad/s.
• Controller: duck-card-loop/1. The row keeps a board: every card targets an absolute pose on it, so a card corrects what the last one left. Walk cards steer onto the line; Turn cards stop short by their measured coast, settle and trim. Dead bands from system identification: forward starts near 0.20 to 0.25 m/s, back near -0.35 m/s, turning near 0.5 to 0.625 rad/s.

Pre-registered criteria

Written into docs/PACKS.md on 2026-10-15, before any result was in, and not changed since. A condition passes only if every criterion passes.

• Position: at least 95% of rows end within 10 cm of the target.
• Heading: at least 95% of rows end within 15° of the target.
• Falls: the upper end of the Wilson 95% interval on the fall rate is at most 1% (with zero falls, that takes 381 rows).
• Time: no Walk card over 8 s and no Turn card over 6 s.
• Ending badly: every row that fails says so to the child.

Every share on this page is the observed share with its Wilson 95% interval in brackets.

Conditions

Evidence ladder

1. Rung 1: Pollen's duck-sim, Monte Carlo. Every criterion in every condition, for the pre-registered rows and separately for rows with a Walk back card. Not passed.
2. Rung 2: a second, clean machine. The same commit and conditions from a fresh checkout on different hardware, matching every verdict. Run on 2026-10-16 on Hugging Face Jobs (x86, AMD EPYC); 38 of 40 verdicts matched, so not reproduced.
3. Rung 3: a real duck on a taped 30 cm grid. Studio's Measure my duck protocol, tape-measured and filmed. Not started; any fall sends the work back to rung 1.

Only when all three pass does a card pack turn closed-loop cards on for real ducks.

Results

Pre-registered rows, duck-card-loop/1. First machine: a Raspberry Pi 5. Second machine: Hugging Face Jobs cpu-upgrade. 455 rows per condition on each.

Two of the 40 rung 1 verdicts differ, both heading at randomized 10 Hz: this row, and the same criterion for Walk back rows below. The intervals overlap; a repeat on the first machine got 93.8%. We read it as a true rate near 95%, on the line, and count it as failing. Falls were 0 in every condition on both machines, and every Time and Ending badly criterion passed.

Not adopted: the mat as a compass (duck-card-loop/4)

A taped grid on the floor, 30 cm squares, sits outside the duck and does not drift. After each Walk or Turn card the duck takes one frame from its head camera, keeps the dark tape pixels whose rays meet the floor 0.2 to 1 m ahead, and finds the small turn and shift that put them back on the grid. Heading comes back to the nearest quarter turn and position to the nearest square. In the simulator the mat is rendered from the duck's true pose, and each condition adds errors in what the duck believes about its camera (pitch, roll, mount, height, brightness, noise). Pre-registered with /3's rule before any row ran.

Run 1 (4,800 rows on Hugging Face Jobs), against /1 on the same machine type:

• Long rows' position at randomized 10 Hz: 36.7% to 80.8%; randomized 33.3% to 82.5%; stressed 10 Hz 0% to 46.7%. Their heading error stopped growing: median 3.6° against 8.6°.
• Falls in every condition, from none: 146 in all.
• Nominal, pre-registered position: 99.1% to 93.7%.

Rejected by its own rule. The machine looks like the cause. This is not proven. 170 rows stopped because the duck's pose stopped arriving for half a second; the /1 run had 2. 17 falls came before the mat had changed anything. The host's load median was 260, against 21 for /3's run. The mat was rendered in software, which by default takes every core for each frame, beside four simulated ducks on 8 vCPUs. On the Pi, with a GPU, development rows had no falls.

The repeat was pre-registered after run 1 and before any row of it. It changes only the instrument: one render thread, at a lower priority than the duck. The seeds, code, criteria and rule are run 1's. It counts only if 10 or fewer of its 4,800 rows lose the pose. If more do, it is called inconclusive, and the next idea starts from /1.

The repeat (4,800 rows, load median 16, 0.05 s a frame):

• 11 rows lost the pose, one more than allowed: inconclusive by its own gate.
• Had it counted, the rule would still reject it. 8 falls in all, from none, fail three verdicts /1 passed. 3 of the 8 came on a row's first card, before any look.
• What the mat did: long rows' position, randomized 33.3% to 95.8% (passes), randomized 10 Hz 36.7% to 92.5%, stressed 10 Hz 0% to 66.7%. Every row under randomization now passes position and heading (75.1% to 97.5%). Nominal is unchanged.
• Still failing: stressed 10 Hz everywhere. Walk back position under stress got worse (79.9% to 68.0%).

So it is not adopted and not shipped. The code stays, unused unless a spec asks for it. A later candidate that keeps the mat will be pre-registered as new, with fresh seeds.

A control: the shipped loop, run again

The mat's repeat failed on 8 falls, judged against one run of the shipped loop (/1) that had none. So /1 was run again on the same machine type with fresh seeds, 4,800 rows, its rule written down first: 3 or more falls would mean the falls criterion cannot tell a candidate from the simulator's own falls.

• /1 fell 2 times. Under the line of 3, so by the rule the falls were the mat's, and the mat stays not adopted. No rows lost the pose.
• But two runs of /1 differ in seven verdicts. Two of them are falls: one fall in 510 rows is enough to fail the 1% bound.
• Rows are not independent. The simulator draws a condition's physics (friction, mass, heading drift) once per session of 12 rows, so 400 rows are about 34 ducks. Intervals that count rows as independent, including the ones on this page, are too narrow.

A secondary analysis was written then, before any row of the next candidate was read. It compares a candidate with both /1 runs pooled, by a permutation test over sessions, with all 80 "worse" tests held to 5% together (Holm). Run on /1 against /1 it finds nothing, both ways. Looking back, as exploration only, with verdicts unchanged: calibration is still worse with no drift at all. The mat's falls are within /1's own. Its real cost is Walk back rows under stress (32% of rows miss on position, against 20%), and its gain on long rows is clear (93% against 31%). A later mat candidate has that to fix, not falls.

Kept, then built on: merge cards (duck-card-loop/5)

Long rows fail because the sensed pose drifts with time, so a row that takes less time should drift less. Consecutive Walk cards run as one walk, up to a cap, and two quarter turns the same way run as one half turn. The board, the cards and every sentence the duck says are unchanged. Pre-registered before any row, with the same rule as before.

Choosing the cap, on development rows only (60 rows a condition, not a result): long rows at randomized 10 Hz passed on position 8 of 15 with no merging, 11 with two walks at a time, 7 with three, 2 with four, and 3 with every walk merged. Every stop lets the next card re-aim at the board; one long walk loses that. The cap is 2.

The evaluation (4,800 rows on Hugging Face Jobs, fresh seeds, no rows lost the pose):

• No verdict that passed for /1 fails, and long rows' position at randomized 10 Hz rose from 36.7% to 41.7%. By its rule it replaces /1 as the base for the next idea.
• Smaller than predicted: 45 to 55% was expected. Long rows took 6 to 9% less time, not a fifth to a third. The stops a merged run saves are a small part of a row.
• Against both /1 runs, counted by session, it is worse nowhere, and its rise is not clear of noise (p 0.18).
• Walk back rows at randomized 10 Hz now pass on position (93.1% to 95.7%). Stressed 10 Hz still fails, so rung 1 still fails. 1 fall in 4,800 rows.

The shipped app keeps /1. /5 is where the next candidate starts, not a change to what Studio runs.

The new base: the mat on the nest (duck-card-loop/7)

The mat as a compass (above) lifted long rows more than anything else, and was not adopted because Walk back rows under stress got worse. Read after the control, those rows split on one thing: the simulated camera's tilt error. With the camera tilted 2° one way, 89 of 214 rows missed by more than 10 cm; tilted 2° the other way, 1 of 142. A camera tilted other than the duck believes stretches the floor along its view, so the mat's fix lands a few centimetres short along the walk. That was a hypothesis, so it was tested as one.

• The change. /6, with /4's mat added unchanged, and one new thing: the mat calibrates the camera. At each look, the duck tries tilt corrections from -4° to +4° and keeps the one under which the tape it sees lies best on lines one square apart. Scores add up over a chapter's looks. The method's settings were chosen on synthetic pictures before any simulated row.
• The mat lies under the nest, so the mat also sees, and corrects, the child's imperfect re-seat.
• Pre-registered before any evaluation row, with predictions: long rows at randomized 10 Hz above 90%; Walk back under stress no worse than /6's 73.8%, the tilt split gone; the tilt taken out within half a degree in most chapters; the stressed condition still failing somewhere. Decided by the test that counts sessions, against /6.

The evaluation (4,800 rows on Hugging Face Jobs, fresh seeds, no row lost the pose):

• Long rows at randomized 10 Hz rose from 63.3% to 97.5% (p 0.0002, counted by session), worse nowhere after correcting for 80 tests. By its rule it replaces /6 as the base.
• Walk back rows under stress: position from 73.8% to 89.2%. The tilt split is gone: tilted one way, 20 of 180 rows missed by more than 10 cm; the other way, 26 of 220.
• The tilt it took out has no bias (mean error at most 0.1°), and is within half a degree in 92% of chapters under randomization, 79% under stress.
• Each look costs time: a long row takes about 5 s longer.
• Stressed 10 Hz still fails, so rung 1 still fails.

Not in the app yet. The app ships /6. /7 needs the duck's head camera read in the app, the mat reader ported and held to the same test vectors as the sim's, and a printed mat. None of this shows a real lens, real light or a real taped mat.

Kept, then built on: a nest and chapters (duck-card-loop/6)

The first three fixes tried to make the duck drift less. This one accepts the drift and shortens what drifts. A long challenge plays in chapters, and between chapters the child puts the duck back on a nest square, as they already seat it at the start. Drift then builds over a chapter, not a whole row.

• Chapter length was set by a rule, not by hand. In /5's rows, rows of 7 cards or fewer landed 95.0% of the time, and rows of 8 or fewer 93.8%. So a chapter is at most 7 cards, split as evenly as possible.
• The child's re-seat is not perfect in the sim. Each re-seat after the first places the duck off the nest: about 1.5 cm and 3° under randomized conditions, 3 cm and 5° under stress. The board believes the duck is on the nest. Each chapter is scored against the true nest, and a row passes only if every chapter does.
• Decided by the test that counts sessions, against /5: the first idea judged that way, declared before any row was read.

The evaluation (4,800 rows on Hugging Face Jobs, fresh seeds, 1 row lost the pose):

• Long rows at randomized 10 Hz rose from 41.7% to 63.3% (p 0.026, counted by session), and it is worse nowhere after correcting for 80 tests. By its rule it replaces /5 as the base.
• Under what was predicted: 70 to 85% was expected. The re-seat costs about as much as a chapter's own drift: in long rows at randomized 10 Hz, the second chapter failed 32 times and the first 15.
• The old rule would have rejected it, on eight Walk back verdicts. No Walk back row has more than 7 cards, so none was split: those rows ran /5's code again on fresh seeds. Their 9 falls, against /5's 0, are the same noise the control found.
• Stressed 10 Hz still fails, so rung 1 still fails.

In the app, from build 87. Both card readers now carry /6. In Packs, a row of more than 7 Walk and Turn cards plays in chapters: the board rings each nest and pauses on it, and Run on the duck stops at each nest until someone presses "The duck is on the nest". Closed-loop cards stay off on real ducks, so a real duck's cards are still timed; there the nest is simply where the child puts the duck back on the board, a game rule that nothing here measured. Measure my duck still runs each card on its own. The sim cannot show that a child will re-seat a duck this well, or will want to.

Rejected candidates

The Walk lead (duck-card-loop/2)

duck-card-loop/2 gave Walk cards the lead Turn cards have: stop short by the measured coast, settle, and lead later Walk cards by the row's mean coast. Its parameters were chosen on development seeds only, then evaluated once on fresh seeds (4,800 rows), on both machines. Against /1 on the same kind of machine:

• Walk back position at randomized 10 Hz: FAIL 93.1% to PASS 96.3%.
• Walk back heading, randomized: PASS 98.3% to FAIL 92.5%.
• Long rows' position at randomized 10 Hz: 36.7% to 20.0%.
• Falls: 11 on the overloaded first machine, 0 in 4,800 rows on the second.

It trades one failing criterion for another and makes long rows worse. Not adopted.

Calibrate at every stop (duck-card-loop/3)

The duck stands still for 5 s before a row and between cards; the yaw rate it reports while standing is taken as drift (a pooled least-squares slope over every still window) and taken out of heading and position. Pre-registered with its decision rule before any row ran; chosen on development seeds; evaluated once on fresh seeds, 4,800 rows on Hugging Face Jobs. The rule: it replaces /1 only if no verdict /1 passed becomes a fail and long rows' position at randomized 10 Hz rises. Against /1 on the same machine type:

• Long rows' position at randomized 10 Hz: 36.7% to 45.8% (rises).
• Stressed 10 Hz, pre-registered position: 84.4% to 92.9%; long rows 0% to 30.8%.
• Falls: none to 1 in 510 (randomized), and 2, 4 and 1 in 493 Walk back rows. Six of the ten came on a row's first card, just after the stand.
• Every row at nominal, where no drift is injected: position 99.0% to 81.5%; long rows 98.3% to 45.0%.

Rejected by its own rule. The rows say why: the estimate follows the injected drift but sits about 0.15°/s low in every condition, nominal included. A standing duck's yaw really moves (the policy creeps or settles), and a sensor inside the duck cannot tell that from drift. Subtracting it for the whole row turns a small true creep into a heading error that grows with time. Heading needs a reference outside the duck, which is the next idea.

Why long rows fail

The controller already aims every card at an absolute place on the board, so its own errors do not stack. What stacks is what the duck believes about itself. A heading sensor drifting at up to 0.5°/s is off by up to 30° after a minute, and a minute is about a ten-card row; odometry that reads 5% long or short is off by 5% of every metre walked. Neither is visible to the controller. The enemy is elapsed time and distance, not the number of cards, and no per-card tuning removes a wrong sensor reading.

What is being tried next

Each is a new controller version, held to the same criteria above, evaluated on fresh seeds on a quiet machine, and published here whether it passes or not. One at a time, in this order:

1. Calibrate at every stop. Tried and rejected: a standing duck is not still enough for its own sensor to measure drift.
2. The mat as a compass. Grid lines seen by the duck's camera fix heading to the nearest quarter turn, and position to the nearest square. Not adopted: long rows far better, but Walk back under stress got worse (control).
3. Spend less time. Merge consecutive Walk cards into one walk and two quarter turns into a half turn, so a row takes less time to drift. Kept as the base: a small gain, 37% to 42% on long rows.
4. A reset in the game. A nest square where the child re-seats the duck, and long challenges split into chapters no longer than the length that passes. The new base, then built on: long rows 42% to 63%.
5. The mat on the nest. Back to the compass, now that the nest gives it a place to lie, with the camera's tilt measured by the mat. The new base in the sim: long rows 63% to 97.5%. Next, porting it to the app.

Separately, Measure my duck now has a 30 second Stand still step that records how much a real duck's heading drifts standing still. If that argues for a gentler simulated condition, the condition is added and pre-registered as new; the conditions above are not edited.

Reproduce it

From a fresh checkout of duck-studio on Linux (x86 or arm64), with nothing else running:

sh cardreader/sim/validate.sh
python3 cardreader/sim/mc/mc_summary.py cardreader/sim/results/raw \
  cardreader/sim/results/raw-acceptance cardreader/sim/results/raw-back

validate.sh builds Pollen's stack at pinned commits, runs system identification and every Monte Carlo batch, and writes HOST.txt (machine, commit, Pollen SHAs) and a per-minute load log. Each session's randomness is seeded from the batch seed and the rows already done, so a rerun is a fresh statistical sample, not a replay: compare verdicts and intervals, not rows. A loaded machine makes simulated bodies fall behind real time and fall; check the logs for "behind real time" before trusting a fall.

Limits

• No real duck has run a closed-loop card yet.
• duck-sim's body drives MuJoCo position servos with gains fitted to the real servos, not the BAM servo model v8 was trained with, which Pollen's infer_policy.py uses.
• Real floors, real IMU drift and real foot slip appear only through the randomized ranges. The simulated drift is one constant rate for a whole session; a real one wanders.
• Rows within a 12-row session share their physics, so they are not independent samples (control).
• The second machine is shared hardware; its load reading covers the host, not the job. No simulated body logged falling behind real time.

Data and code

• The pre-registration and every result docs/PACKS.md
• Every row: first machine, repeat, and the Hugging Face Jobs run in rung2-hf cardreader/sim/results
• The Walk lead's rows cardreader/sim/results-walkback
• Calibrate at every stop, criterion by criterion cardreader/sim/results-calibrate
• The mat as a compass, the repeat, criterion by criterion cardreader/sim/results-compass
• The mat as a compass, run 1, criterion by criterion cardreader/sim/results-compass
• The control: /1 again, the secondary analysis and its self-test cardreader/sim/results-control
• Merge cards, criterion by criterion, and its secondary analysis cardreader/sim/results-merge
• A nest and chapters, criterion by criterion, and the deciding analysis cardreader/sim/results-nest
• The mat on the nest, criterion by criterion, the tilt it took out, and the deciding analysis cardreader/sim/results-mat-nest
• The controller cardloop.py
• A draft proposal to Pollen: walk by distance, turn by angle not filed

Questions, a second reproduction or a correction: email or open an issue on duck-studio.