...
Back

What a Health Check Should Actually Check

Most health endpoints answer a question nobody asked. The useful distinction is between 'is this process alive' and 'should this instance receive traffic', and confusing them causes outages in both directions.

What a Health Check Should Actually Check

What a Health Check Should Actually Check ❤️

The default health endpoint returns 200 OK and the string ok. It proves the process is running and the HTTP server is listening. That is a real fact and it is almost never the fact anyone needs.

The question an orchestrator is actually asking is: should I send this instance a user's request right now? A process can be running and be a terrible answer to that question — still loading configuration, holding a dead database connection, thrashing on memory, or in the middle of shutting down.


Liveness and readiness are different questions

Liveness: is this process broken beyond recovery? A failed liveness check should cause a restart. It must therefore be extremely conservative — check only that the process itself is functional, with no dependencies involved. A liveness check that queries the database will restart every one of your instances during a database blip, turning a degradation into a full outage. This is the single most expensive health-check mistake, and it is common precisely because "check the database" sounds more thorough.

Readiness: should this instance receive traffic right now? A failed readiness check should remove the instance from rotation without killing it. This one may consider dependencies, carefully, and it is the one that should reflect startup state: configuration loaded, caches warmed, connections established, migrations applied.

Collapsing these into one endpoint forces a single answer to two questions with different correct responses.


What readiness should check

Include things that are specific to this instance and whose failure means this instance genuinely cannot serve:

  • Startup is complete. Configuration parsed, required secrets present, any once-per-process initialization finished.
  • Connection pools are established.
  • The instance is not draining. On shutdown, readiness should fail before the server stops accepting connections, so the load balancer stops sending work while in-flight requests finish. Skipping this is the most common cause of errors during otherwise clean deploys.

Exclude things that are shared across all instances. If a dependency is down, every instance fails readiness, every instance leaves rotation, and you have converted a partial failure into a total one — and often taken away the instances that could still serve cached or degraded responses. Shared dependency health belongs in monitoring and alerts, not in the signal that controls routing.


Deep checks belong at a different URL

There is real value in an endpoint that verifies the whole dependency graph — during deploys, in smoke tests, for on-call triage. Just do not let the orchestrator route on it.

/healthz    liveness   — process is functional, no dependencies
/readyz     readiness  — this instance can serve, instance-scoped only
/statusz    diagnostic — full dependency detail, authenticated, not routed on

The diagnostic endpoint should report per-dependency status and latency, and it should be authenticated: it is a precise map of your internal topology.


Two failure modes worth naming

The check that checks nothing. An endpoint that returns a constant string, or worse, a static file served by the reverse proxy in front of the app. It answers 200 while the application behind it is dead. If you cannot say which failure your health check would catch, it is decoration.

The check that is too clever. A readiness probe that itself performs heavy work — a full query, a cache warm, an external call — adds load exactly when the system is struggling, and its own timeout becomes a failure mode. Health checks run constantly; they must be cheap enough that their cost is invisible at steady state.


The generalizable part

Deploy problems that look like networking are frequently readiness problems. "The service came up but requests failed for the first thirty seconds" is not a firewall issue, it is an instance that reported ready before it was. "Requests failed during the deploy" is usually an instance that stopped accepting connections before it left rotation.

Both come from the same root: the signal that controls routing was not actually measuring whether routing would succeed. Getting that one signal honest — instance-scoped, cheap, and false while starting up and while draining — removes a whole category of intermittent deploy failures that are otherwise miserable to diagnose.