You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/04/14 12:48:17 UTC
[GitHub] [apisix] rdouman opened a new issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
rdouman opened a new issue #4048:
URL: https://github.com/apache/apisix/issues/4048
## Issue description
I have a 10 node kubernetes cluster running APISIX as a gateway, We're using etcd 3.4.14 deployed as a cluster via the bitnami helm chart. We're intermittently receiving the errors below on container startup.
2021/04/14 14:09:03 [error] 42#42: *1 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["\/ecom\/*"],"vars":{},"hosts":["dev.url","develop.url"],"id":"350417588708180914","update_time":1618401644,"status":1,"create_time":1618394758,"name":"ecom_dev_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.development.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","POST","PUT","DELETE"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 42#42: *1 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["\/ecom\/*"],"vars":{},"hosts":["uat.url","stage.url"],"id":"350426062628324274","update_time":1618401371,"status":1,"create_time":1618399809,"name":"ecom_uat_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.uat.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","HEAD","POST","PUT","DELETE","OPTIONS","PATCH"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 41#41: *2 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["\/ecom\/*"],"vars":{},"hosts":["dev.url","develop.url"],"id":"350417588708180914","update_time":1618401644,"status":1,"create_time":1618394758,"name":"ecom_dev_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.development.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","POST","PUT","DELETE"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 41#41: *2 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["\/ecom\/*"],"vars":{},"hosts":["uat.url","stage.url"],"id":"350426062628324274","update_time":1618401371,"status":1,"create_time":1618399809,"name":"ecom_uat_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.uat.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","HEAD","POST","PUT","DELETE","OPTIONS","PATCH"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 45#45: *3 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["\/ecom\/*"],"vars":{},"hosts":["dev.url","develop.url"],"id":"350417588708180914","update_time":1618401644,"status":1,"create_time":1618394758,"name":"ecom_dev_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.development.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","POST","PUT","DELETE"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 45#45: *3 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["\/ecom\/*"],"vars":{},"hosts":["uat.url","stage.url"],"id":"350426062628324274","update_time":1618401371,"status":1,"create_time":1618399809,"name":"ecom_uat_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.uat.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","HEAD","POST","PUT","DELETE","OPTIONS","PATCH"]}, context: init_worker_by_lua*
APISIX appears to be listening on the exposed port, but returns 404 route not found whenever trying to access one of the configured end points on containers with the above error in the logs.
## Expected behavior
APIXSIX should start up with no errors in the error logs and successfully route traffic on the exposed routes.
## How to Reproduce
1. Deploy APISIX gateway on kubernetes cluster. Scale the deployment and check the logs.
## Environment
* apisix version (cmd: `apisix version`):
apache/apisix:2.2-alpine
* OS (cmd: `uname -a`):
/etc/os-release
NAME="openSUSE MicroOS"
# VERSION="20210408"
ID="opensuse-microos"
ID_LIKE="suse opensuse opensuse-tumbleweed"
VERSION_ID="20210408"
PRETTY_NAME="openSUSE MicroOS"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:microos:20210408"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:MicroOS"
LOGO="distributor-logo"
* OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`):
* etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API):
3.4.14
* apisix-dashboard version, if have:
apache/apisix-dashboard:2.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] souzens commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
souzens commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-820018017
> > failed to validate the 'vars' expression: rule too short
>
> This message is added in APISIX 2.5, and apisix-dashboard 2.3 is too old to work with APISIX 2.5
the same wrong ..
NOW etcd data is created by apisix-dashboard 2.4 . and After upgrading to apisix 2.5, the current problem is that the existing etcd data can no longer be used, and all of the created routes has been invalid.
according to [https://github.com/apache/apisix/blob/master/docs/zh/latest/admin-api.md](https://github.com/apache/apisix/blob/master/docs/zh/latest/admin-api.md)
vars is still use list type , apisix etcd 2.5 report vars use dict type .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] rdouman commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
rdouman commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-820361055
Hi all
Thanks you for the quick response. I just want to point out that I'm not using APISIX 2.5. I'm using the docker image for APISIX /apache/apisix:2.2-alpine with the apache/apisix-dashboard:2.3 dashboard.
One thing that I've noticed is that the apache/apisix:2.2-alpine has been updated 10 days ago, so my question is has this image perhaps been erroneous updated ?
I've tracked down the intermittent nature of the issue:
Docker images that are working are apache/apisix:2.2-alpine (digest 186f0052e1191)
Docker images that are not working are apache/apisix:2.2-alpine (digest 22a167dfcf31e)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-820228487
BTW, none of the apisix-dashboard releases support APISIX 2.5 now. If you need to use the dashboard, please use APISIX 2.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #4048:
URL: https://github.com/apache/apisix/issues/4048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-819953134
> failed to validate the 'vars' expression: rule too short
This message is added in APISIX 2.5, and apisix-dashboard 2.3 is too old to work with APISIX 2.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-819951301
> _failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/_"],"vars":{},
According to the prompt information here, there seems to be a problem with the `vars` configured in the route. Is the vars expression set in your route `"vars":{}`? Maybe you should provide detailed configuration information for routing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-820227275
Can be solved by applied #4056 or wait for 2.6 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] souzens edited a comment on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
souzens edited a comment on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-820018017
> > failed to validate the 'vars' expression: rule too short
>
> This message is added in APISIX 2.5, and apisix-dashboard 2.3 is too old to work with APISIX 2.5
the same wrong ..
NOW etcd data is created by apisix-dashboard 2.4 . and After upgrading to apisix 2.5, the current problem is that the existing etcd data can no longer be supported, and all of the created routes has been invalid.
according to [https://github.com/apache/apisix/blob/master/docs/zh/latest/admin-api.md](https://github.com/apache/apisix/blob/master/docs/zh/latest/admin-api.md)
vars is still use list type , apisix etcd 2.5 report vars use dict type .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-820084080
Just submit a PR to fix it: https://github.com/apache/apisix/pull/4056
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-820083904
> the current problem is that the existing etcd data can no longer be supported, and all of the created routes has been invalid.
My bad. I underestimate the effect of the additional check.
> vars is still use list type , apisix etcd 2.5 report vars use dict type .
No, the `vars` is still list type. The problem is that we add validation to reject empty table. We also update the dashboard but of course, the existing routes are broken now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #4048: request help: Intermittent failure when starting apisix container on kubernetes.
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #4048:
URL: https://github.com/apache/apisix/issues/4048#issuecomment-819934650
What's your apisix version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org