You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "lkad (via GitHub)" <gi...@apache.org> on 2024/12/26 06:12:00 UTC
[I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
lkad opened a new issue, #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341
### Current Behavior
like #2167 ,i install apsix ingress-controller use etcdserver=true,but somtime ingress-controller can restart ,then i add route or change rout in apisixroute ,it not effect .
install apisix :
ADMIN_API_VERSION=v3
helm install apisix . \
--set service.type=NodePort \
--set ingress-controller.enabled=true \
--create-namespace \
--namespace ingress-apisix \
--set ingress-controller.config.apisix.serviceNamespace=ingress-apisix \
--set ingress-controller.config.apisix.adminAPIVersion=$ADMIN_API_VERSION \
--set ingress-controller.config.kubernetes.enableGatewayAPI=true \
--set dashboard.enabled=true \
--set ingress-controller.config.etcdserver.enabled=true
then add apisixroute
```
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: test-route
namespace: testroute2
spec:
http:
- name: route-1
match:
hosts:
- routetest2cccc.ccc.ccc
paths:
- /*
backends:
- serviceName: httpbin
servicePort: 80
```
```
kind: Deployment
apiVersion: apps/v1
metadata:
name: httpbin
namespace: testroute2
labels:
app: httpbin
annotations:
deployment.kubernetes.io/revision: '1'
kubesphere.io/creator: admin
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
template:
metadata:
creationTimestamp: null
labels:
app: httpbin
annotations:
kubesphere.io/creator: admin
kubesphere.io/imagepullsecrets: '{}'
kubesphere.io/restartedAt: '2024-11-28T00:57:38.581Z'
spec:
containers:
- name: container-cjo9a0
image: mccutchen/go-httpbin
ports:
- name: http-0
containerPort: 80
protocol: TCP
env:
- name: PORT
value: '80'
resources:
limits:
cpu: '2'
memory: 1000Mi
requests:
cpu: 200m
memory: 200Mi
readinessProbe:
httpGet:
path: /
port: 80
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
```
```
kind: Service
apiVersion: v1
metadata:
name: httpbin
namespace: testroute2
labels:
app: httpbin
annotations:
kubesphere.io/creator: admin
spec:
ports:
- name: http-1
protocol: TCP
port: 80
targetPort: 80
selector:
app: httpbin
type: ClusterIP
sessionAffinity: None
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
internalTrafficPolicy: Cluster
```
please add more other apisixroute
then curl ip/get -H "host: routetest2cccc.ccc.ccc" .return 200.
then find the ingress on which node ,one the node exec shell kill the ingress-controller
" ps -elf | grep ingress-controller | awk '{print $4}' | xargs kill "
then change the apisixroute
```
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: test-route
namespace: testroute2
spec:
http:
- name: route-1
match:
hosts:
- routenew.ccc.ccc
paths:
- /*
backends:
- serviceName: httpbin
servicePort: 80
```
curl ip/get -H "host: routenew.ccc.ccc" .return 404.
### Expected Behavior
curl ip/get -H "host: routenew.ccc.ccc" .return 200.
### Error Logs
apisix contoianer log
2024/12/26 06:07:13 [error] 59#59: *3033128 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:13 [error] 51#51: *3033129 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:13 [error] 56#56: *3033130 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:13 [error] 54#54: *3033131 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 49#49: *3033267 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 52#52: *3033268 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 55#55: *3033269 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 53#53: *3033270 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 59#59: *3033271 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 50#50: *3033272 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 56#56: *3033273 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 51#51: *3033274 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:16 [error] 54#54: *3033275 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 49#49: *3033413 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 52#52: *3033414 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 53#53: *3033415 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 55#55: *3033416 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 59#59: *3033417 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 50#50: *3033418 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 56#56: *3033419 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 51#51: *3033420 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2024/12/26 06:07:19 [error] 54#54: *3033421 [lua] config_etcd.lua:193: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
### Steps to Reproduce
like #2167 ,i install apsix ingress-controller use etcdserver=true,but somtime ingress-controller can restart ,then i add route or change rout in apisixroute ,it not effect .
install apisix :
ADMIN_API_VERSION=v3
helm install apisix . \
--set service.type=NodePort \
--set ingress-controller.enabled=true \
--create-namespace \
--namespace ingress-apisix \
--set ingress-controller.config.apisix.serviceNamespace=ingress-apisix \
--set ingress-controller.config.apisix.adminAPIVersion=$ADMIN_API_VERSION \
--set ingress-controller.config.kubernetes.enableGatewayAPI=true \
--set dashboard.enabled=true \
--set ingress-controller.config.etcdserver.enabled=true
then add apisixroute
```
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: test-route
namespace: testroute2
spec:
http:
- name: route-1
match:
hosts:
- routetest2cccc.ccc.ccc
paths:
- /*
backends:
- serviceName: httpbin
servicePort: 80
```
```
kind: Deployment
apiVersion: apps/v1
metadata:
name: httpbin
namespace: testroute2
labels:
app: httpbin
annotations:
deployment.kubernetes.io/revision: '1'
kubesphere.io/creator: admin
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
template:
metadata:
creationTimestamp: null
labels:
app: httpbin
annotations:
kubesphere.io/creator: admin
kubesphere.io/imagepullsecrets: '{}'
kubesphere.io/restartedAt: '2024-11-28T00:57:38.581Z'
spec:
containers:
- name: container-cjo9a0
image: mccutchen/go-httpbin
ports:
- name: http-0
containerPort: 80
protocol: TCP
env:
- name: PORT
value: '80'
resources:
limits:
cpu: '2'
memory: 1000Mi
requests:
cpu: 200m
memory: 200Mi
readinessProbe:
httpGet:
path: /
port: 80
scheme: HTTP
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
```
```
kind: Service
apiVersion: v1
metadata:
name: httpbin
namespace: testroute2
labels:
app: httpbin
annotations:
kubesphere.io/creator: admin
spec:
ports:
- name: http-1
protocol: TCP
port: 80
targetPort: 80
selector:
app: httpbin
type: ClusterIP
sessionAffinity: None
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
internalTrafficPolicy: Cluster
```
please add more other apisixroute
then curl ip/get -H "host: routetest2cccc.ccc.ccc" .return 200.
then find the ingress on which node ,one the node exec shell kill the ingress-controller
" ps -elf | grep ingress-controller | awk '{print $4}' | xargs kill "
then change the apisixroute
```
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: test-route
namespace: testroute2
spec:
http:
- name: route-1
match:
hosts:
- routenew.ccc.ccc
paths:
- /*
backends:
- serviceName: httpbin
servicePort: 80
```
curl ip/get -H "host: routenew.ccc.ccc" .return 404.
### Environment
- APISIX Ingress controller version (run `apisix-ingress-controller version --long`)
apache/apisix-ingress-controller:1.8.0
- Kubernetes cluster version (run `kubectl version`)
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8
- OS version if running APISIX Ingress controller in a bare-metal environment (run `uname -a`)
apache/apisix:3.8.1-debian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "lkad (via GitHub)" <gi...@apache.org>.
lkad commented on issue #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341#issuecomment-2562200643
curl "http://10.204.222.7:9180/apisix/admin/plugin_configs/11a33fd6" -H "X-API-KEY: the route is added .
export ETCDCTL_ENDPOINTS=10.204.222.7:12379
export ETCDCTL_API=3
etcdctl get /apisix --prefix=true the route is add to etcd .but use curl ,don't take effect , return 404 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "lkad (via GitHub)" <gi...@apache.org>.
lkad commented on issue #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341#issuecomment-2572815765
in apisix pod run this command
```
"sed 's/if tonumber(res.result.header.revision) > self.prev_index/if true or tonumber(res.result.header.revision) > self.prev_index/g' -i /usr/local/apisix/apisix/core/config_etcd.lua"
```
then run :
```
apisix restart
```
then i kill apisix-ingress ,the route change take effect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "lkad (via GitHub)" <gi...@apache.org>.
lkad commented on issue #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341#issuecomment-2562287793
when i change apisixroute .
apisix add record to etcd,and notifiy to apisix,but apisix note apply to route
```
2024/12/26 07:06:27 [info] 56#56: *15617 [lua] config_etcd.lua:120: produce_res(): append res: {
result = {
events = { {
kv = {
create_revision = "1031",
key = "/apisix/upstreams/5019bcf9",
mod_revision = "1033",
value = {
desc = "Created by apisix-ingress-controller, DO NOT modify it manually",
id = "5019bcf9",
labels = {
["managed-by"] = "apisix-ingress-controller"
},
name = "testroute2_httpbin_80",
nodes = { {
host = "192.168.70.252",
port = 80,
weight = 100
},
<metatable> = <1>{}
},
scheme = "http",
type = "roundrobin"
}
},
prev_kv = {
create_revision = "1031",
key = "/apisix/upstreams/5019bcf9",
mod_revision = "1031",
value = {
desc = "Created by apisix-ingress-controller, DO NOT modify it manually",
id = "5019bcf9",
labels = {
["managed-by"] = "apisix-ingress-controller"
},
name = "testroute2_httpbin_80",
nodes = { {
host = "192.168.70.252",
port = 80,
weight = 100
},
<metatable> = <table 1>
},
scheme = "http",
type = "roundrobin"
}
}
}, {
kv = {
create_revision = "1032",
key = "/apisix/routes/48d337a6",
mod_revision = "1034",
value = {
desc = "Created by apisix-ingress-controller, DO NOT modify it manually",
hosts = { "routetest3.ccc.ccc",
<metatable> = <table 1>
},
id = "48d337a6",
labels = {
["managed-by"] = "apisix-ingress-controller"
},
name = "testroute2_test-route_route-1",
plugin_config_id = "6058ee0d",
plugins = {
["host-rate-limit"] = {
capacity = 1000,
interval = 1000,
key = "http_test",
key_type = "var",
lock_enable = true,
max_wait = 1000,
quantum = 1000,
rejected_code = 503
}
},
upstream_id = "5019bcf9",
uris = { "/*",
<metatable> = <table 1>
}
}
},
prev_kv = {
create_revision = "1032",
key = "/apisix/routes/48d337a6",
mod_revision = "1032",
value = {
desc = "Created by apisix-ingress-controller, DO NOT modify it manually",
hosts = { "routetest2cccc.ccc.ccc",
<metatable> = <table 1>
},
id = "48d337a6",
labels = {
["managed-by"] = "apisix-ingress-controller"
},
name = "testroute2_test-route_route-1",
plugin_config_id = "6058ee0d",
plugins = {
["host-rate-limit"] = {
capacity = 1000,
interval = 1000,
key = "http_test",
key_type = "var",
lock_enable = true,
max_wait = 1000,
quantum = 1000,
rejected_code = 503
}
},
upstream_id = "5019bcf9",
uris = { "/*",
<metatable> = <table 1>
}
}
}
} },
header = {
revision = "1034"
},
watch_id = "42"
}
}, err: nil, context: ngx.timer
2024/12/26 07:06:31 [info] 51#51: *16770 client closed connection while waiting for request, client: 127.0.0.1, server: 0.0.0.0:9180
```
below function
```
-- append res to the queue and notify pending watchers
local function produce_res(res, err)
if log_level >= NGX_INFO then
log.info("append res: ", inspect(res), ", err: ", inspect(err))
end
insert_tab(watch_ctx.res, {res=res, err=err})
for _, sema in pairs(watch_ctx.sema) do
sema:post()
end
table.clear(watch_ctx.sema)
end
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "lkad (via GitHub)" <gi...@apache.org>.
lkad commented on issue #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341#issuecomment-2562207046
I suspect that when etcdserver=true, it starts an in-memory etcd. After the ingress-controller restarts, the etcdserver data is lost. At the same time, after restart, the ingress-controller will batch refresh the routes in APISIX. However, since the route data in APISIX's memory is newer, the index or version refreshed to etcd is smaller, causing APISIX to not apply the latest routes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341#issuecomment-2781811602
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2341: bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller
URL: https://github.com/apache/apisix-ingress-controller/issues/2341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341#issuecomment-2856761483
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: When etcdserver=true, newly added ApisixRoutes don't take effect after restarting the ingress-controller [apisix-ingress-controller]
Posted by "lkad (via GitHub)" <gi...@apache.org>.
lkad commented on issue #2341:
URL: https://github.com/apache/apisix-ingress-controller/issues/2341#issuecomment-2562293555
i think is this code : when route etcd version < route in apsix mem version . not do update .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org