You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "xinlifoobar (via GitHub)" <gi...@apache.org> on 2026/04/01 11:51:37 UTC
[I] bug: health check keeps probing stale upstream nodes after nodes update (checker cache not invalidated) [apisix]
xinlifoobar opened a new issue, #13141:
URL: https://github.com/apache/apisix/issues/13141
### Current Behavior
When updating an upstream's nodes (static upstream, no service discovery), APISIX health check continues probing nodes that no longer exist in the upstream configuration.
Specifically:
After removing a node from upstream.nodes,
APISIX still performs active health checks on the removed node
This results in repeated error logs like:
2026/03/31 11:37:15 [error] 60#60: *156956195 [lua] healthcheck.lua:1383: log(): [healthcheck] (upstream#/apisix/upstreams/00000000000000085365) failed to receive status line from '10.xx.xx.xx(10.xx.xx.xx:9358)': timeout, context: ngx.timer
The stale node is not present in the current upstream config, but still exists in the health checker.
### Expected Behavior
When upstream nodes are updated:
Removed nodes should also be removed from the health checker
Health check should only probe current upstream nodes
No stale endpoints should remain in the checker
### Error Logs
[lua] healthcheck.lua:1383: log(): [healthcheck]
(upstream#/apisix/upstreams/xxxx)
failed to receive status line from '10.xx.xx.xx:9358': timeout, context: ngx.timer
### Steps to Reproduce
1. Run APISIX (e.g., via Docker)
2. Create an upstream with multiple nodes and enable active health check:
```
{
"nodes": {
"10.0.0.1:8080": 1,
"10.0.0.2:8080": 1
},
"checks": {
"active": {
"type": "http",
"http_path": "/health",
"interval": 1
}
}
}
```
3. Wait for health check to start
4. Update upstream and remove one node, e.g.:
{
"nodes": {
"10.0.0.1:8080": 1
}
}
Observe:
- APISIX still performs health check on 10.0.0.2:8080
- Error logs continue for the removed node
5. Reload APISIX:
`apisix reload
`
→ Issue disappears
### Environment
- APISIX version (run `apisix version`): 3.14.1-ubuntu
- Operating system (run `uname -a`): Linux core-gateway-sh-hd-volc-apisix-6576dd94db-v5vqk 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
nginx version: openresty/1.27.1.2
built by gcc 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04)
built with OpenSSL 3.4.1 11 Feb 2025
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_RUNTIME_VER=1.3.2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl3/include' --add-module=../ngx_devel_kit-0.3.3 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.28 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.37 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.20 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.16 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/li
b -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl3/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl3/lib' --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../ngx_multi_upstream_module-1.3.2 --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../apisix-nginx-module-1.19.2 --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../apisix-nginx-module-1.19.2/src/stream --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../apisix-nginx-module-1.19.2/src/meta --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../wasm-nginx-module-0.7.0 --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.iBso3vORdI/openresty-1.27.1.2/../lua-resty-events-0.2.0 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_mod
ule --with-http_v3_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):9.7.3
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: health check keeps probing stale upstream nodes after nodes update (checker cache not invalidated) [apisix]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan closed issue #13141: bug: health check keeps probing stale upstream nodes after nodes update (checker cache not invalidated)
URL: https://github.com/apache/apisix/issues/13141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: health check keeps probing stale upstream nodes after nodes update (checker cache not invalidated) [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #13141:
URL: https://github.com/apache/apisix/issues/13141#issuecomment-4174241439
Hi @xinlifoobar, thanks for your report. There are two questions that require your further confirmation:
1. What is your current worker_processes configuration? We need to confirm the number of worker processes when the problem occurred.
2. Could you please change the log level to info and then post the logs before and after the problem occurred? This would help us further investigate the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: health check keeps probing stale upstream nodes after nodes update (checker cache not invalidated) [apisix]
Posted by "hanzhenfang (via GitHub)" <gi...@apache.org>.
hanzhenfang commented on issue #13141:
URL: https://github.com/apache/apisix/issues/13141#issuecomment-4686861553
Hi @Baoyuantop, @nic-6443 , I built a small repro that makes the stale health-check target visible through the removed node's access log. The removed node stays alive, so the signal is not a timeout; it is simply whether APISIX keeps sending `GET /health` to a node that is no longer in the upstream.
One thing I could not tell from the issue is how the upstream was updated. That matters here. A full `PUT` replacement and a partial `PATCH` have different semantics.
## Files
### docker-compose.yaml
```yaml
name: issue-13141
services:
etcd:
image: bitnamilegacy/etcd:3.6.4
container_name: issue-13141-etcd
restart: "no"
environment:
ALLOW_NONE_AUTHENTICATION: "yes"
ETCD_ADVERTISE_CLIENT_URLS: http://etcd:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ports:
- "13144:2379"
apisix:
image: ${APISIX_IMAGE:-apache/apisix:3.14.1-debian}
container_name: issue-13141-apisix
restart: "no"
depends_on:
- etcd
- node1
- node2
volumes:
- ./config.yaml:/usr/local/apisix/conf/config.yaml:ro
ports:
- "13141:9080"
- "13142:9180"
- "13143:9090"
node1:
image: nginx:alpine
container_name: issue-13141-node1
restart: "no"
environment:
NODE_NAME: node1
volumes:
- ./mock-nginx.conf:/etc/nginx/templates/default.conf.template:ro
networks:
default:
ipv4_address: 172.31.41.11
node2:
image: nginx:alpine
container_name: issue-13141-node2
restart: "no"
environment:
NODE_NAME: node2
volumes:
- ./mock-nginx.conf:/etc/nginx/templates/default.conf.template:ro
networks:
default:
ipv4_address: 172.31.41.12
networks:
default:
ipam:
config:
- subnet: 172.31.41.0/24
```
### config.yaml
```yaml
apisix:
node_listen:
- 9080
enable_admin: true
nginx_config:
error_log_level: info
worker_processes: 1
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
admin_listen:
ip: 0.0.0.0
port: 9180
admin_key:
- name: admin
key: issue-13141-admin-key
role: admin
allow_admin:
- 0.0.0.0/0
etcd:
host:
- http://etcd:2379
control:
ip: 0.0.0.0
port: 9090
```
### mock-nginx.conf
```nginx
log_format issue13141 '$time_iso8601 $hostname $request_method $uri $status';
server {
listen 8080;
access_log /dev/stdout issue13141;
error_log /dev/stderr info;
location = /health {
add_header X-Issue-Node "${NODE_NAME}" always;
return 200 "${NODE_NAME} healthy\n";
}
location / {
add_header X-Issue-Node "${NODE_NAME}" always;
return 200 "${NODE_NAME} response\n";
}
}
```
## Steps
Start:
```sh
docker compose up -d --force-recreate --remove-orphans
docker compose exec -T apisix apisix version
```
Create an upstream with two nodes and active health checks:
```sh
curl -sS -X PUT http://127.0.0.1:13142/apisix/admin/upstreams/1 \
-H 'X-API-KEY: issue-13141-admin-key' \
-H 'Content-Type: application/json' \
-d '{
"type": "roundrobin",
"nodes": {
"172.31.41.11:8080": 1,
"172.31.41.12:8080": 1
},
"checks": {
"active": {
"type": "http",
"http_path": "/health",
"healthy": {
"interval": 1,
"successes": 1
},
"unhealthy": {
"interval": 1,
"http_failures": 1
}
}
}
}'
curl -sS -X PUT http://127.0.0.1:13142/apisix/admin/routes/1 \
-H 'X-API-KEY: issue-13141-admin-key' \
-H 'Content-Type: application/json' \
-d '{
"uri": "/hc-cache",
"upstream_id": "1"
}'
```
Warm up and confirm both nodes are checked:
```sh
curl -sS http://127.0.0.1:13141/hc-cache
sleep 5
docker logs --since 10s issue-13141-node1 2>&1 | grep 'GET /health'
docker logs --since 10s issue-13141-node2 2>&1 | grep 'GET /health'
```
Remove `node2` from the upstream:
```sh
curl -sS -X PUT http://127.0.0.1:13142/apisix/admin/upstreams/1 \
-H 'X-API-KEY: issue-13141-admin-key' \
-H 'Content-Type: application/json' \
-d '{
"type": "roundrobin",
"nodes": {
"172.31.41.11:8080": 1
},
"checks": {
"active": {
"type": "http",
"http_path": "/health",
"healthy": {
"interval": 1,
"successes": 1
},
"unhealthy": {
"interval": 1,
"http_failures": 1
}
}
}
}'
```
Confirm the node is really gone from the stored upstream:
```sh
curl -sS http://127.0.0.1:13142/apisix/admin/upstreams/1 \
-H 'X-API-KEY: issue-13141-admin-key'
```
Hit the route again so APISIX instantiates the checker for the new upstream version, then count health checks after the update:
```sh
curl -sS http://127.0.0.1:13141/hc-cache
MARK=$(date -u +%Y-%m-%dT%H:%M:%SZ)
sleep 8
echo "after update marker: $MARK"
echo "node1 health probes after update:"
docker logs --since "$MARK" issue-13141-node1 2>&1 | grep -c 'GET /health' || true
echo "node2 health probes after update:"
docker logs --since "$MARK" issue-13141-node2 2>&1 | grep -c 'GET /health' || true
```
Expected healthy behavior:
```text
node1 health probes after update: non-zero
node2 health probes after update: 0
```
Bug signal:
```text
node2 health probes after update: non-zero
```
If `node2` is still being probed after the Admin API shows only `172.31.41.11:8080` in the upstream, the health checker target set is stale in the worker and was not reconciled when `upstream.nodes` changed.
In my local run with `apache/apisix:3.14.1-debian`, this repro did show stale probing when the upstream was replaced with `PUT`:
```text
before update:
node1 -> GET /health once per second
node2 -> GET /health once per second
after PUT with only 172.31.41.11:8080 and another route hit:
node1 -> GET /health once per second
node2 -> GET /health once per second
```
I also tested several update methods:
```text
admin PUT replace upstream body:
stored config no longer contained node2
node2 still received health checks
=> stale checker reproduced
admin PATCH partial nodes object:
stored config still contained node2
node2 still received health checks
=> not a stale-checker signal; node2 was not removed
admin PATCH node2 weight 0:
stored config still contained node2 with weight 0
node2 still received health checks
=> node remains a health-check target
direct etcd v3 put full replacement:
stored config no longer contained node2
node2 still received health checks
=> stale checker reproduced
```
One caveat I found while testing: `PATCH` with a partial `nodes` object merges the object and does not remove omitted nodes. In that case the Admin API response still contains the old node, so continued health checks are expected.
I also checked the timing. With full `PUT` replacement, stale health checks to the removed node were visible immediately after the update, but they did not persist indefinitely in this local lab:
```text
PUT then immediate route hit:
first 8s window: node1_health=8, node2_health=8
next 20s window: node1_health=21, node2_health=4
PUT long window:
first 30s window: node1_health=29, node2_health=11
next 60s window: node1_health=59, node2_health=0
next 120s window: node1_health=118, node2_health=0
```
So this local repro shows a stale checker transition window after a full upstream replacement, but it does not by itself explain a case where a removed node is still checked days later. For that longer-lived symptom, I would first verify whether the old node still exists in the effective data-plane upstream config, whether only some APISIX data-plane instances missed the update, or whether the health checks are coming from another upstream/service/route that still references the node.
**I think we should troubleshoot how to "remove" the old upstream nodes.**
```text
Admin API PUT:
full replacement; omitted nodes should disappear from stored config
Admin API PATCH:
partial merge; omitted nodes may remain in stored config
Dashboard/control-plane update:
needs checking at the generated Admin API / etcd / DP config layer, because the UI action may be implemented as either replace or merge
direct etcd write:
depends on whether the full upstream object is replaced or a stale object is written back
decoupled CP/DP sync:
needs checking on each data plane, because one DP may still hold an older upstream version even if the control plane has the new object
```
The key diagnostic split is: if the removed node is still present in the effective upstream config for the APISIX instance sending `/health`, the behavior is explained by update semantics or config propagation. If the effective upstream config no longer contains the node but that same APISIX instance still sends active checks to it, then the problem is stale health-checker state. For multi-DP or decoupled deployments, this check should be done per data-plane instance, not only on the control-plane object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org