The Retry That Made the Outage Worse 🔁

A dependency gets slow. Your client times out and retries. So does every other client, at the same moment, because they all failed at the same moment. The dependency now receives more traffic than it did when it was healthy, while being less able to serve it. It gets slower. More timeouts fire. More retries arrive.

Nobody wrote a loop. The loop is emergent, and it is the default behavior of naive retry logic under correlated failure.

Three properties turn a retry into an amplifier

Synchronization. Clients that fail together retry together. A fixed delay preserves the correlation forever — the herd just moves in lockstep one second later. This is why jitter is not a refinement but the core of the fix: randomizing each client's delay is what breaks the herd into a spread.

Multiplication. Retries at multiple layers compose multiplicatively. Three attempts in the HTTP client, inside three attempts in the service wrapper, inside a job runner that retries the whole task, is twenty-seven requests for one logical operation. Each layer looks reasonable on its own. Nobody sees the product, because no single file contains it.

No concept of hopelessness. A client that retries a dependency which has been failing for ten minutes is not being resilient, it is being an attacker. Retries help with transient faults — a dropped packet, a brief leader election, one unlucky instance. They do nothing at all for a dependency that is down, and they actively prevent its recovery by consuming the capacity it needs to come back.

What correct actually looks like

Exponential backoff with full jitter. Not "add a little randomness" — sample the whole interval:

delay = random.uniform(0, min(cap, base * 2 ** attempt))

This spreads a synchronized herd across the window rather than moving it intact.

Retry in exactly one layer. Pick the layer that knows what the operation means and can decide whether repeating it is safe, and make every other layer pass failures through. This is the single highest-value change in most codebases, and it usually means deleting retry logic rather than adding it.

Retry only what is retryable. A timeout or a 503, yes. A 400, a 401, a validation error — never; the answer will not change and you are spending the dependency's capacity to be told so again. Rate limiting is its own case: a 429 means stop, and if it carries a Retry-After, that value is an instruction, not a suggestion.

Give up. A budget — attempts, or better, total elapsed time — after which you fail and say so. Failing fast and visibly is more useful than retrying invisibly forever, because it surfaces the problem instead of converting it into unexplained slowness.

Break the circuit. After enough consecutive failures, stop sending entirely for a cooldown, then let one probe through. This is the piece that lets the dependency recover, and it is the piece most often missing, because it requires admitting that the right move is to stop trying.

Two cases worth calling out

Rate limits are not transient faults. When an upstream enforces a per-account limit, concurrency does not help — it is the thing being limited. The only shape that works is serialize, persist each success immediately, and resume from where you stopped. A batch job that retries the whole batch from the start after hitting a limit will never finish, and will look like it is working the entire time.

Background probes need budgets too. Periodic health or enrichment checks feel free because each one is small. At enough concurrency they will exhaust the local socket or file-descriptor limit and produce resource-exhaustion errors that look nothing like a network problem. If something runs on a timer against many targets, it needs a concurrency cap as much as any user-facing path does.

The question to ask in review

When you see a retry, ask: if every client did this at the same time, what would the dependency experience?

If the answer is "more load than when it was healthy," the retry is not resilience. It is a small denial-of-service attack you have scheduled against yourself, and it will fire on exactly the day you can least afford it.