You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "shreemaan-abhishek (via GitHub)" <gi...@apache.org> on 2026/06/18 09:22:45 UTC
[PR] fix(control-api): exclude deleted upstream nodes from health check API [apisix]
shreemaan-abhishek opened a new pull request, #13572:
URL: https://github.com/apache/apisix/pull/13572
### Description
The health check Control API (`GET /v1/healthcheck` and `/v1/healthcheck/{src_type}/{src_id}`) reads its node list straight from the health checker's shared dict via `healthcheck.get_target_list()`. That dict is keyed only by the upstream id, not by the node set, so when a node is removed from an upstream its target can linger there:
- during the `delayed_clear` grace window after the checker is recreated, and
- potentially indefinitely, since the cleanup sweep only runs for a checker that is present in the periodic-lock-holding worker's in-memory checker set.
As a result the Control API keeps reporting nodes that are no longer part of the upstream, which is what #13385 reports.
This patch reconciles the reported nodes against the upstream's current configuration in `extra_checker_info`, dropping any target whose `ip:port` is not in the current node set (taking the active-check port override into account). Matching is fail-open: if the configured node set cannot be determined, the original list is returned unchanged so a live node is never hidden.
Fixes #13385
### Reproduce / verify
Added `t/control/healthcheck-stale-nodes.t`: create an upstream with two nodes and an active health check, start the checker, delete one node via the Admin API, then query the Control API. Before the fix the deleted node is still returned; after the fix only the remaining node is reported.
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(control-api): exclude deleted upstream nodes from health check API [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13572:
URL: https://github.com/apache/apisix/pull/13572#discussion_r3445414906
##########
docs/en/latest/control-api.md:
##########
@@ -134,7 +134,7 @@ Each of the returned objects contain the following fields:
* name: resource id, where the health checker is reporting from.
* type: health check type: `["http", "https", "tcp"]`.
-* nodes: target nodes of the health checker.
+* nodes: target nodes of the health checker. Nodes removed from the upstream are not reported, even if their checker state has not been cleaned up yet.
Review Comment:
```suggestion
* nodes: target nodes of the health checker. When APISIX can determine the current upstream node set from the configuration, nodes that have been removed from that set are not reported (even if their checker state has not been cleaned up yet).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(control-api): exclude deleted upstream nodes from health check API [apisix]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13572:
URL: https://github.com/apache/apisix/pull/13572#discussion_r3435386037
##########
apisix/control/v1.lua:
##########
@@ -69,6 +69,54 @@ function _M.schema()
end
local healthcheck
+
+
+-- targets the checker registers for this upstream, keyed host:port
+-- (active-check port overrides the node port)
+local function get_configured_targets(up_conf)
+ local nodes = up_conf and up_conf.nodes
+ if type(nodes) ~= "table" then
+ return nil
+ end
+
+ local active = up_conf.checks and up_conf.checks.active
+ local override_port = active and active.port
+ local targets = core.table.new(0, #nodes)
+
Review Comment:
`core.table.new(0, #nodes)` uses the length operator for preallocation, which will be 0 when `nodes` is a hash table (the common `{ "ip:port": weight }` form). This doesn’t break correctness, but it defeats the intended preallocation and can cause extra rehashing for large upstreams; use `core.table.nkeys(nodes)` for the map case.
##########
docs/en/latest/control-api.md:
##########
@@ -134,7 +134,7 @@ Each of the returned objects contain the following fields:
* name: resource id, where the health checker is reporting from.
* type: health check type: `["http", "https", "tcp"]`.
-* nodes: target nodes of the health checker.
+* nodes: target nodes of the health checker. Nodes removed from the upstream are not reported, even if their checker state has not been cleaned up yet.
Review Comment:
This sentence reads like an unconditional guarantee, but the implementation is explicitly fail-open (it returns the original node list unchanged if the configured node set can’t be determined, e.g. discovery-based upstreams without a `nodes` table). Consider softening the wording to reflect that filtering applies when APISIX can reconcile against the current upstream nodes configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(control-api): exclude deleted upstream nodes from health check API [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13572:
URL: https://github.com/apache/apisix/pull/13572#discussion_r3445413883
##########
apisix/control/v1.lua:
##########
@@ -69,6 +71,54 @@ function _M.schema()
end
local healthcheck
+
+
+-- targets the checker registers for this upstream, keyed host:port
+-- (active-check port overrides the node port)
+local function get_configured_targets(up_conf)
+ local nodes = up_conf and up_conf.nodes
+ if type(nodes) ~= "table" then
+ return nil
+ end
+
+ local active = up_conf.checks and up_conf.checks.active
+ local override_port = active and active.port
+ local targets = core.table.new(0, #nodes)
Review Comment:
```suggestion
local n = core.table.isarray(nodes) and #nodes or core.table.nkeys(nodes)
local targets = core.table.new(0, n)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(control-api): exclude deleted upstream nodes from health check API [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on PR #13572:
URL: https://github.com/apache/apisix/pull/13572#issuecomment-4764506189
closing in favor of: https://github.com/apache/apisix/pull/13572#pullrequestreview-4540907975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(control-api): exclude deleted upstream nodes from health check API [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek closed pull request #13572: fix(control-api): exclude deleted upstream nodes from health check API
URL: https://github.com/apache/apisix/pull/13572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org