You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "solosky (via GitHub)" <gi...@apache.org> on 2026/04/23 03:54:35 UTC
[I] bug: Health check state lost and checker not working after upstream node changes [apisix]
solosky opened a new issue, #13282:
URL: https://github.com/apache/apisix/issues/13282
### Current Behavior
When upstream nodes change (e.g., Kubernetes pod scaling, service discovery update, or DNS resolution change), the health checker has two critical issues:
1. **Health check status lost**: Previously detected unhealthy nodes reset to healthy after node changes
2. **Health check not running**: There is a probability that the health checker stops checking after node changes
### Root Cause
APISIX uses a **full destroy-and-rebuild** strategy for health checkers when upstream nodes change. The core flow is:
1. Node change → `_nodes_ver` increments → `resource_version` changes
2. `fetch_checker()` detects version mismatch → adds to `waiting_pool`, returns `nil` (no checker during this period)
3. Timer (1s interval) destroys old checker → calls `delayed_clear()` to **clear all health status from shared dict**
4. Creates brand new checker → **all nodes start as healthy**
### Impact
- Traffic routed to unhealthy nodes during the window between checker rebuild and next active check cycle
- Removed nodes remain in the health checker's target list, consuming resources and potentially affecting health check results
- In high-frequency node change scenarios, the checker may never be successfully created due to version race conditions
### Environment
- APISIX version: 3.16.0
- lua-resty-healthcheck-api7: 3.2.1-0
### Steps to Reproduce
1. Create a route with health check enabled and multiple upstream nodes
2. Wait for one node to be detected as unhealthy
3. Add a new node to the upstream (or trigger service discovery update)
4. Observe that the previously unhealthy node resets to healthy
5. Check health checker target list — removed nodes may still be present
### Expected Behavior
1. Existing nodes should retain their health status (healthy/unhealthy) when nodes are added/removed
2. Removed nodes should be properly cleaned up from the health checker
3. Health checking should not have gaps during node changes
### Suggested Fix
Implement **incremental target update** instead of full rebuild:
1. When nodes change but `checks` config remains the same, only add/remove targets on the existing checker
2. Use `target.hostname` from `get_target_list()` when calling `remove_target()` to ensure the correct target is matched
3. Only do full rebuild when `checks` configuration changes
Key changes in `healthcheck_manager.lua`:
- Add `update_checker_targets()`: incrementally adds new targets and removes stale ones
- Add `checks_config_equal()`: compares checks config to decide incremental vs full rebuild
- Fix `remove_target()` hostname: use stored `target.hostname` instead of `checks.active.host`
- Save `checks` config in working pool for later comparison
### Expected Behavior
1. Existing nodes should retain their health status (healthy/unhealthy) when nodes are added/removed
2. Removed nodes should be properly cleaned up from the health checker
3. Health checking should not have gaps during node changes
### Error Logs
_No response_
### Steps to Reproduce
1. Create a route with health check enabled and multiple upstream nodes
2. Wait for one node to be detected as unhealthy
3. Add a new node to the upstream (or trigger service discovery update)
4. Observe that the previously unhealthy node resets to healthy
5. Check health checker target list — removed nodes may still be present
### Environment
Environment
APISIX version: 3.16.0
lua-resty-healthcheck-api7: 3.2.1-0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Health check state lost and checker not working after upstream node changes [apisix]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan closed issue #13282: bug: Health check state lost and checker not working after upstream node changes
URL: https://github.com/apache/apisix/issues/13282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Health check state lost and checker not working after upstream node changes [apisix]
Posted by "arunmat27 (via GitHub)" <gi...@apache.org>.
arunmat27 commented on issue #13282:
URL: https://github.com/apache/apisix/issues/13282#issuecomment-4706520863
Hi all - we are currently evaluating APISIX for production and hit this issue in one of our staging environment. Is there a workaround this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Health check state lost and checker not working after upstream node changes [apisix]
Posted by "vortegatorres (via GitHub)" <gi...@apache.org>.
vortegatorres commented on issue #13282:
URL: https://github.com/apache/apisix/issues/13282#issuecomment-4439725859
Hi! We are hitting the same issue in production (APISIX 3.16.0 standalone, `lua-resty-healthcheck-api7 3.2.1-0`, upstreams using `service_name + discovery_type: "kubernetes"`): long-lived APISIX pods keep health-checking Pod IPs that are no longer present in either the standalone admin `/configs` output or the `/v1/discovery/kubernetes/dump` output, until the pod is restarted. We previously reported the same underlying behaviour from the ingress-controller side in apache/apisix-ingress-controller#2708, where it surfaced as stale IPs lingering in `upstream_addr`; with active health checks enabled it now also produces `lua-resty-events` queue-overflow floods on the affected pods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Health check state lost and checker not working after upstream node changes [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #13282:
URL: https://github.com/apache/apisix/issues/13282#issuecomment-4302286864
Hi @solosky, could you provide more detailed steps to reproduce the issue? Please include the specific APISIX configuration and routes configuration details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Health check state lost and checker not working after upstream node changes [apisix]
Posted by "solosky (via GitHub)" <gi...@apache.org>.
solosky commented on issue #13282:
URL: https://github.com/apache/apisix/issues/13282#issuecomment-4302455711
# Reproducing Health Check State Lost After Upstream Node Changes
## Prerequisites
- minikube
- helm
- kubectl
- Docker (or use minikube docker-env)
## Step 1: Start Minikube and Install APISIX
```bash
# Start minikube
minikube start --cpus=4 --memory=8192 --driver=docker
# Add APISIX Helm chart repo
helm repo add apisix https://charts.apiseven.com
helm repo update
# Install APISIX and APISIX Ingress Controller
helm install apisix apisix/apisix \
--namespace ingress-apisix \
--create-namespace \
--set gateway.type=NodePort \
--set gateway.http.nodePort=30080 \
--set etcd.persistence.enabled=false \
--set ingress-controller.enabled=true
# Wait for all pods ready
kubectl -n ingress-apisix wait --for=condition=Ready pods --all --timeout=120s
```
## Step 2: Build and Load the Test Image
The test server is a simple Node.js static file server that returns `X-Pod-IP` and `X-Pod-Name` headers on every response. This allows us to identify which pod served each request.
**server.js**
```js
const http = require('http');
const fs = require('fs');
const path = require('path');
const os = require('os');
const PORT = process.env.PORT || 8080;
const STATIC_DIR = path.join(__dirname, 'static');
function getLocalIP() {
const interfaces = os.networkInterfaces();
for (const name of Object.keys(interfaces)) {
for (const iface of interfaces[name]) {
if (iface.family === 'IPv4' && !iface.internal) {
return iface.address;
}
}
}
return '127.0.0.1';
}
function getPodName() {
return process.env.HOSTNAME || 'unknown';
}
const POD_IP = getLocalIP();
const POD_NAME = getPodName();
const MIME_TYPES = {
'.html': 'text/html',
'.css': 'text/css',
'.js': 'application/javascript',
'.json': 'application/json',
'.png': 'image/png',
'.jpg': 'image/jpeg',
'.gif': 'image/gif',
'.svg': 'image/svg+xml',
'.ico': 'image/x-icon',
'.txt': 'text/plain',
};
function getMimeType(filePath) {
const ext = path.extname(filePath).toLowerCase();
return MIME_TYPES[ext] || 'application/octet-stream';
}
const server = http.createServer((req, res) => {
const startTime = Date.now();
const requestIP = req.socket.remoteAddress || 'unknown';
const userAgent = req.headers['user-agent'] || 'unknown';
res.setHeader('X-Pod-IP', POD_IP);
res.setHeader('X-Pod-Name', POD_NAME);
let filePath = req.url === '/' ? '/index.html' : req.url;
filePath = path.normalize(filePath).replace(/^(\.\.[\/\\])+/, '');
const fullPath = path.join(STATIC_DIR, filePath);
fs.readFile(fullPath, (err, data) => {
let statusCode = 200;
if (err) {
if (err.code === 'ENOENT') {
statusCode = 404;
res.writeHead(404, { 'Content-Type': 'text/plain' });
res.end('404 Not Found');
} else {
statusCode = 500;
res.writeHead(500, { 'Content-Type': 'text/plain' });
res.end('500 Internal Server Error');
}
} else {
const mimeType = getMimeType(fullPath);
res.writeHead(200, { 'Content-Type': mimeType });
res.end(data);
}
const elapsed = Date.now() - startTime;
const time = new Date().toISOString();
console.log(`[${time}] "${req.method} ${req.url}" ${statusCode} ${elapsed}ms ${requestIP} "${userAgent}"`);
});
});
server.listen(PORT, '0.0.0.0', () => {
console.log(`Server running on port ${PORT}`);
console.log(`Pod IP: ${POD_IP}`);
});
```
**Dockerfile**
```dockerfile
FROM node:20-alpine
WORKDIR /app
COPY server.js .
ADD static ./static
EXPOSE 8080
CMD ["node", "server.js"]
```
**static/ files** — Create the following files in a `static/` directory:
- `static/index.html` — any HTML content
- `static/healthcheck.html` — any content (used by APISIX active health check)
- `static/readiness.html` — any content (used by Kubernetes readiness probe)
- `static/liveness.html` — any content (used by Kubernetes liveness probe)
Build and load the image into minikube:
```bash
# Build inside minikube's Docker daemon
eval $(minikube docker-env)
docker build -t httpbin-test:latest .
```
## Step 3: Deploy the Test Application (2 Replicas)
**httpbin-deployment.yaml**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin
labels:
app: httpbin
spec:
replicas: 2
selector:
matchLabels:
app: httpbin
template:
metadata:
labels:
app: httpbin
spec:
containers:
- name: httpbin
image: httpbin-test:latest
imagePullPolicy: Never
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /readiness.html
port: 8080
initialDelaySeconds: 3
periodSeconds: 5
livenessProbe:
httpGet:
path: /liveness.html
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
```
```bash
kubectl apply -f httpbin-deployment.yaml
# Wait for both pods to be Running and Ready
kubectl get pods -l app=httpbin -w
# You should see 2/2 READY, e.g.:
# httpbin-5f55f9686c-abcde 1/1 Running 0 30s
# httpbin-5f55f9686c-fghij 1/1 Running 0 30s
```
## Step 4: Create Service, ApisixRoute, and ApisixUpstream
**httpbin-resources.yaml**
```yaml
apiVersion: v1
kind: Service
metadata:
name: httpbin-svc
spec:
selector:
app: httpbin
ports:
- port: 8080
targetPort: 8080
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: httpbin-route
spec:
http:
- name: httpbin
match:
hosts:
- httpbin.local
paths:
- "/*"
backends:
- serviceName: httpbin-svc
servicePort: 8080
---
apiVersion: apisix.apache.org/v1
kind: ApisixUpstream
metadata:
name: httpbin-svc
spec:
healthCheck:
active:
type: http
httpPath: /healthcheck.html
healthy:
interval: 2s
httpStatuses:
- 200
successes: 1
unhealthy:
interval: 2s
httpFailures: 1
httpStatuses:
- 40
- 500
- 502
- 503
- 504
tcpFailures: 1
timeouts: 1
```
```bash
kubectl apply -f httpbin-resources.yaml
```
## Step 5: Verify Health Check is Working
Get the APISIX NodePort:
```bash
APISIX_PORT=$(kubectl get svc -n ingress-apisix apisix-gateway -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}")
APISIX_IP=$(minikube ip)
echo "APISIX endpoint: http://${APISIX_IP}:${APISIX_PORT}"
# Send requests and observe X-Pod-Name header to see which pod serves each request
curl -s -H "Host: httpbin.local" http://${APISIX_IP}:${APISIX_PORT}/ -D -
# You should see X-Pod-Name and X-Pod-IP headers in the response
# Trigger health check by sending a few requests
for i in $(seq 1 10); do
curl -s -H "Host: httpbin.local" http://${APISIX_IP}:${APISIX_PORT}/ -o /dev/null -w "%{http_code}\n"
done
```
## Step 6: Identify a Target Pod and Set Up Monitoring
Pick one of the two pods and start a monitoring loop:
```bash
# List pods
kubectl get pods -l app=httpbin
# Example output:
# httpbin-5f55f9686c-abcde 1/1 Running 0 5m
# httpbin-5f55f9686c-fghij 1/1 Running 0 5m
# Replace <POD_NAME> with one of the pod names from above
TARGET_POD=<POD_NAME> # e.g., httpbin-5f55f9686c-abcde
# Start monitoring loop — this prints a line whenever the TARGET_POD serves a request
while true; do
curl -s -I -H "Host: httpbin.local" http://${APISIX_IP}:${APISIX_PORT}/ | grep "${TARGET_POD}";
done
```
You should see the target pod name printed periodically as requests are load-balanced to it.
## Step 7: Make the Target Pod Unhealthy
Open a shell into the target pod and remove/rename the health check file:
```bash
# Exec into the target pod
kubectl exec -it ${TARGET_POD} -- sh
# Inside the pod: rename the healthcheck file so APISIX active check gets a 404
mv /app/static/healthcheck.html /app/static/healthcheck.html.bak
# Exit the pod
exit
```
Wait for APISIX to detect the node as unhealthy (within ~4 seconds based on the check interval). The monitoring loop should **stop printing output** — confirming no traffic is being routed to the unhealthy pod.
```bash
# Verify via APISIX health check API (optional)
kubectl exec -n ingress-apisix <APISIX_POD> -- curl -s http://127.0.0.1:9180/apisix/admin/upstreams -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
```
## Step 8: Scale Up Replicas to 3 (Trigger Node Change)
In a new terminal:
```bash
kubectl scale deployment httpbin --replicas=3
```
## Step 9: Observe the Bug
Go back to the monitoring loop terminal. **You will see the target pod name start printing again**, meaning traffic is being routed to the unhealthy pod despite its health check still failing.
```bash
# Example of buggy output:
# (before scaling: no output — correctly excluded from load balancing)
# httpbin-5f55f9686c-abcde <-- BUG: traffic resumed to unhealthy pod after scale-up
# httpbin-5f55f9686c-abcde
# httpbin-5f55f9686c-abcde
```
This confirms the health check status was **lost** during the upstream node change.
## Expected Behavior
After scaling to 3 replicas:
- The existing unhealthy pod should **remain unhealthy** — no traffic should be routed to it
- The 2 healthy pods (original healthy + new pod) should handle all traffic
- Only after the unhealthy pod's healthcheck file is restored should traffic resume to that pod
## Root Cause
When upstream nodes change (triggered by the scale-up), APISIX:
1. Detects the `resource_version` change
2. **Destroys** the old health checker and calls `delayed_clear()` to clear all health status from the shared dict
3. **Creates** a new checker where all nodes start as healthy by default
This means previously detected unhealthy nodes reset to healthy, causing traffic to be routed to them until the next active health check cycle discovers they are unhealthy again.
Additionally, in the incremental update path, `remove_target()` may silently fail because the `hostname` parameter used for removal does not match the one used during `add_target()`. This causes stale targets to remain in the checker even after nodes are removed from the upstream configuration.
## Environment
- APISIX: apache/apisix:3.16.0-ubuntu
- APISIX Ingress Controller: apache/apisix-ingress-controller:2.0.1
- lua-resty-healthcheck-api7: 3.2.1-0
- Kubernetes: minikube (any recent version)
apisix and apisix ingress controller installed via helm
```
helm repo add apisix https://apache.github.io/apisix-helm-chart
helm repo update
helm install apisix apisix/apisix --create-namespace --namespace apisix
helm install apisix-dashboard apisix/apisix-dashboard --create-namespace --namespace apisix
helm install apisix-ingress-controller apisix/apisix-ingress-controller --namespace apisix --create-namespace
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org