Launching Something You Cannot Undo 🚦

Nearly everything we deploy can be rolled back. That single fact shapes every habit we have: ship small, watch the graphs, revert if it looks wrong. It is a good system and it is entirely built on reversibility.

Then you hit an operation that does not have that property — a live payment path, a destructive migration, a record written somewhere permanent, a first real-money transaction — and every habit that made you fast becomes a liability. Not because the habits are bad, but because they assume an undo that no longer exists.

---

Move the safety earlier

With reversible work, safety sits after the action: deploy, observe, revert. With irreversible work there is nothing after, so all of the safety has to sit before. In practice that means three things.

Make the irreversible step require an explicit, out-of-band approval. Not a code review — a separate affirmative act at the moment of execution. Ours is a build substitution that must be passed explicitly; without it the pipeline refuses to proceed to the irreversible stage. The important property is not the specific mechanism but that the default path cannot reach the irreversible action. Someone has to say yes, on purpose, in a way that cannot happen by rerunning a command from history.

Verify the target before acting on it. The single most valuable check is confirming that what you are about to write to is what you think it is: the right environment, the right account, the right identifier — read back from the live system, not from the config file you are reading. Config describes intent; the readback describes reality, and the gap between them is where the expensive mistakes live.

Read back after writing. Write, then read the value from the authoritative source and compare it to what you intended. A write that reports success is not evidence that the correct value is now stored — it is evidence that the request was accepted. For anything you cannot redo, that distinction is the whole game.

---

Gates must not be bypassable in a hurry

A gate exists precisely for the moment when someone is under pressure and certain they are right. If it can be skipped by adding a flag, it will be skipped exactly then.

What has worked for us:

• The gate fails closed. Missing approval means stop, not warn. An approval that defaults to true when unset is not a gate.
• The approval names what is being approved. "Yes, deploy" is weak; "yes, this specific irreversible operation, in this environment" is a statement someone can actually be responsible for.
• The gate is tested, and the test is mutation-verified. Break the gate deliberately and confirm the test fails. A gate test that passes against a broken gate is worse than none, because it manufactures confidence.
• Nobody bypasses it on their own judgment, including the people who wrote it. If it is wrong, fix the gate — do not route around it once and promise to fix it later.

---

Kill switches are a phase, not a design

Before an irreversible launch, kill switches are correct: a flag that disables the new path, a cap on exposure, a hard-coded refusal for the real environment. They let you ship the code before you ship the behavior.

They are also temporary by construction, and the discipline is in removing them. A kill switch left in place after launch is not a safety feature — it is an unexercised branch that nobody tests, gradually becoming a way to break the system by accident.

So removing them is its own change, reviewed as carefully as adding them, with the same question asked in reverse: what breaks if this is now always on? Ours came out in a dedicated commit that did nothing else, which is the shape that lets a reviewer actually check it.

---

The part that generalizes past launches

This all sounds specific to high-stakes operations, but the underlying habit is small and worth having everywhere:

Before any action, know whether it is reversible. If it is not, say so out loud.

Most of the damage from irreversible operations does not come from taking them carelessly. It comes from not noticing they were irreversible — treating a permanent write like a deploy, because it looked like a deploy, because the tooling made it look like a deploy.

The gate, the readback, the explicit approval: those are all mechanisms for making the category visible at the moment it matters. The mechanism is negotiable. The noticing is not.