[PR] fix(healthcheck): cold-start readiness gate for lazily-created, unprobed checkers [apisix]

["aanogueira (via GitHub)" <gi...@apache.org>]

aanogueira opened a new pull request, #13897:
URL: https://github.com/apache/apisix/pull/13897

   ## Problem
   
   A freshly created health-check target defaults to `internal_health = healthy` with zero probes (`add_target`'s hardcoded `is_healthy=true`), so a pod that restarts while its backend is already unhealthy briefly routes real traffic to it until enough active probes correct the target's state. Separately, checker creation itself is entirely lazy -- seeded only by `fetch_checker()` on the live request path -- so an idle-but-critical upstream with no prior traffic would never get a checker built at all ahead of a readiness check.
   
   There is currently no way to tell "healthy" (a real check passed) apart from "healthy" (the zero-probe default) from outside the checker, so nothing can gate readiness on "has this actually been checked yet."
   
   ## Changes
   
   - `apisix/healthcheck_manager.lua`: adds two accessors for a readiness plugin to use --
     - `ensure_checker(resource_path)`: proactively seeds a checker for a resource even with zero prior traffic, reusing the existing `timer_create_checker` construction path. Also resolves domain-name upstream nodes via `parse_domain_in_up` up front -- that resolution otherwise only happens on the live request path, so a checker built ahead of traffic would start probing under an unresolved identity and get silently rebuilt (wiping its probe count) the moment real traffic first resolved the domain.
     - `is_resource_probed(resource_path)`: true only once every target of the resource's checker has had enough real active-check attempts for its state to have actually converged -- a single attempt is not always enough: with e.g. `unhealthy.http_failures = 2` configured, `internal_health` only converges after two consecutive attempts. Computes the required attempt threshold from the checker's own config (`max(unhealthy.http_failures, .tcp_failures, .timeouts, healthy.successes)`) and delegates to a companion module function, `resty.healthcheck.all_targets_probed(name, shm_name, min_attempts)`, proposed as a separate change against lua-resty-healthcheck-api7 (the vendored library this repo depends on): a per-target probe-attempt counter in shm, incremented each time an active check is actually dispatched for a target (success, failure, or timeout all count -- attempted, not "healthy"), queryable from any worker.
   - `t/node/healthcheck-fresh-node-default-healthy.t`: new tests covering lazy checker creation (`fetch_checker` returns `false` until the next timer tick), `ensure_checker` building a checker with zero prior traffic, `all_targets_probed` flipping only after a real probe, and the multi-attempt threshold behavior specifically (stays `false` after 1 attempt when `min_attempts=2`, flips `true` only after the 2nd).
   
   Validated end to end in a local kind cluster: pod-restart-while-unhealthy shows zero leaked requests before or after the readiness transition.
   
   ## Dependency note
   
   The probe-counter shm mechanism lives in the vendored `lua-resty-healthcheck-api7` library, not this repo. The library-side change is being proposed as a standalone PR against api7/lua-resty-healthcheck.
   
   ## Testing
   
   - `t/node/healthcheck-fresh-node-default-healthy.t` (this repo)
   - Local kind end-to-end trial: pod restart while backend already unhealthy, comparing leaked traffic and the readiness transition, stock vs. patched


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(healthcheck): cold-start readiness gate for lazily-created, unprobed checkers [apisix]

["aanogueira (via GitHub)" <gi...@apache.org>]

aanogueira closed pull request #13897: fix(healthcheck): cold-start readiness gate for lazily-created, unprobed checkers
URL: https://github.com/apache/apisix/pull/13897


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org