The toy after a lost round on a walnut desk beside a keyboard and a mug: the six slabs are down and the red SEV-1 flag is up

One Commit Took Down Prod

A desk toy of the oldest story in software: press Enter, six servers fall, and a red SEV-1 flag goes up.

Ready to ship.

The CI spinner waits on its landing inside the rack cage. Six slabs stand in a row behind it, and the SEV-1 flag lies flat.

Press Enter.

The red Enter key on the little laptop is a lever: pressing it tips the landing, and the spinner starts rolling down its fin.

CI is running.

Nothing drives it but gravity. The spinner takes about half a second to reach the first slab, and that is the on-call's window.

It hits DEPLOY.

At 0.52 s the spinner knocks DEPLOY into LB, LB into RACK, then DB, API and the tall PROD.

Prod is down.

PROD lands on the trigger at the far end, and the red SEV-1 flag swings up to stand level.

Stand it up, lay it down.

Stand the toy on its laptop end and lay it flat again. Every slab, the spinner, the key and the flag fall back to the start.

Or the on-call wins.

Push ROLLBACK while the spinner is still rolling and a door swings across its path. It stops short of DEPLOY, and prod stays up.

6 slabs, one stack.

Each slab carries the name of a layer, raised on the rail beneath it. They stand shortest first, so every fall lands on something taller.

  1. DEPLOY1.1 inThe release. The spinner hits it first.
  2. LB1.2 inThe load balancer, taken down by the release.
  3. RACK1.4 inThe hardware everyone forgot about.
  4. DB1.6 inIt was always going to be the database.
  5. API1.7 inEvery client starts timing out here.
  6. PROD2.3 inRed-topped and tallest. When it lands, SEV-1 goes up.

Up close

Made for a developer's desk: 12.5 in long and 2.5 oz, it sits below the monitor where the one who pushes and the one on call can both reach it.

The toy on a light oak desk after a round, a hand resting beside it and a mug behind
Someone pushed on a Friday. Mine lives next to the keyboard.
The rack cage from behind, the CI spinner's dotted disk inside it
The spinner rolls inside its own rack cage, so it never leaves the toy.
The SEV-1 flag standing after PROD fell onto its trigger
PROD down, flag up. You can see it from across the room.
The toy set up on a wooden shelf beside a small succulent in lamp light
Between rounds it sits on the shelf, already set up for the next one.
After a lost round, from behind: five slabs leaning in a row, PROD flat on the trigger, SEV-1 up
A lost round. Each slab leans on the next, right down the line.

Half a second to roll back.

The clock starts when Enter goes down. The spinner's roll sets the deadline, not how hard anyone pushes, so the on-call races the toy's own timing and nothing else.

Window to roll back Chain falling

  1. Enter goes down. The spinner leaves its landing.

  2. A typical person has only just started to move.

  3. A finger already resting on ROLLBACK has the door shut.

  4. Last moment the door can close in front of the spinner.

  5. The spinner hits DEPLOY. Nothing stops the chain now.

  6. PROD lands on the trigger. SEV-1 is up.

The deadline moves only with the table: from 0.45 s to 0.48 s across a 1° tilt either way. Play with a finger on the knob and the on-call usually wins; play with hands flat on the desk and it is a real race.

A saved round seen from behind: the ROLLBACK knob pushed in, every slab still standing
A saved round: ROLLBACK pushed in, the spinner held at the door, every slab still up.

One Commit Took Down Prod

10 moving parts on one frame, all in the box.

6
slabs, DEPLOY to PROD
1
CI spinner in its rack cage
1
red Enter key on a little laptop
1
ROLLBACK knob and its door
1
SEV-1 flag
Size
12.5 × 1.4 × 2.3 in
Weight
2.5 oz
Material
PLA in light grey, red at the top
Players
1–2: one pushes, one is on call, or one hand each
Power
None. Gravity and a finger; no batteries, magnets or screws
Reset
Stand it on its laptop end, then lay it down
Labels
9 raised on the rail, from git push --force to PROD
The toy set up and ready, seen from behind the PROD end

Good to know

Does it need batteries?

No. The spinner rolls by gravity and everything else moves because a slab or a finger pushes it. There are no batteries, magnets, motors or screws.

Can I play on my own?

Yes. Press Enter with one hand and try to reach ROLLBACK with the other, or just knock prod over for the satisfaction. With two players, one pushes and one is on call, then you swap.

Can the on-call really win?

Yes, if they are ready. The door has to close within about 0.46 s of the press. A typical reaction takes 0.27 s, so a finger resting on the knob usually saves prod, and a hand flat on the desk makes it a race.

How do I set it up again?

Stand the toy on its laptop end and lay it back down. Every slab stands up again, the spinner returns to its landing, the Enter key resets and the flag folds flat.

What is it made of?

PLA, light grey with red on the Enter key, the ROLLBACK knob, PROD's top and the SEV-1 flag. Nothing falls off or flies off the desk.