You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/06/24 07:32:41 UTC
[GitHub] [apisix] Gerrylinux opened a new issue, #7322: The worker synchronizes etcd data. When more than 3 routes are created at the same time, the nginx worker cannot obtain the latest etcd data
Gerrylinux opened a new issue, #7322:
URL: https://github.com/apache/apisix/issues/7322
### Description
apisix route yaml file name test.yaml
```
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: testsssss
namespace: default
spec:
http:
- name: "be"
match:
hosts:
- 'a.testsssss.com'
paths:
- "/*"
backends:
- serviceName: nginx-v1
servicePort: 80
priority: 5
- name: "be1"
match:
hosts:
- b.testsssss.com
paths:
- "/*"
backends:
- serviceName: nginx-v1
servicePort: 80
- name: "be2"
match:
hosts:
- 'c.testsssss.com'
paths:
- "/*"
backends:
- serviceName: nginx-v1
servicePort: 80
- name: "be3"
match:
hosts:
- d.testsssss.com
paths:
- "/*"
backends:
- serviceName: nginx-v1
servicePort: 80
```
Added debug logging in apisix/core/config_etcd.lua 351-356
```
348 if not res then
349 return false, err
350 end
351 if res[1] and res[1].value and res[1].value.hosts then
352 local host = res[1].value.host or res[1].value.hosts[1]
353 if ngx.re.find(host, [[^[a-z]\.testsssss\.com$]], "jo") then
354 log.error("new data coming ", ngx.worker.id(), " pid ", ngx.worker.pid() , " host ", host)
355 end
356 end
357
358 local res_copy = res
359 -- waitdir will return [res] even for self.single_item = true
360 for _, res in ipairs(res_copy) do
```
The number of workers is 4
Log output after an Apisix route is created
```
2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host a.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host a.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 50#50: *203538 [lua] config_etcd.lua:354: sync_data(): new data coming nil pid 50 host a.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 44#44: *205783 [lua] config_etcd.lua:354: sync_data(): new data coming 0 pid 44 host a.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 47#47: *202984 [lua] config_etcd.lua:354: sync_data(): new data coming 3 pid 47 host a.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host b.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host b.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host c.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host c.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host d.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host d.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 44#44: *205783 [lua] config_etcd.lua:354: sync_data(): new data coming 0 pid 44 host b.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 47#47: *202984 [lua] config_etcd.lua:354: sync_data(): new data coming 3 pid 47 host b.testsssss.com, context: ngx.timer
2022/06/24 07:13:46 [error] 50#50: *203538 [lua] config_etcd.lua:354: sync_data(): new data coming nil pid 50 host b.testsssss.com, context: ngx.timer
```
The third and fourth routes created are not synchronized and exist in the ETCD data
Is that a problem? Or am I using it the wrong way, and so far I have both environments that can reproduce.
### Environment
- APISIX version (run `apisix version`): :2.11.0
- Operating system (run `uname -a`): k8s v1.22.7
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`): 2.11.0
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): v3.5.4
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] whioue commented on issue #7322: The worker synchronizes etcd data. When more than 3 routes are created at the same time, the nginx worker cannot obtain the latest etcd data
Posted by GitBox <gi...@apache.org>.
whioue commented on issue #7322:
URL: https://github.com/apache/apisix/issues/7322#issuecomment-1214581652
> ### 描述
> apisix 路由 yaml 文件名 test.yaml
>
> ```
> apiVersion: apisix.apache.org/v2beta3
> kind: ApisixRoute
> metadata:
> name: testsssss
> namespace: default
> spec:
> http:
> - name: "be"
> match:
> hosts:
> - 'a.testsssss.com'
> paths:
> - "/*"
> backends:
> - serviceName: nginx-v1
> servicePort: 80
> priority: 5
> - name: "be1"
> match:
> hosts:
> - b.testsssss.com
> paths:
> - "/*"
> backends:
> - serviceName: nginx-v1
> servicePort: 80
> - name: "be2"
> match:
> hosts:
> - 'c.testsssss.com'
> paths:
> - "/*"
> backends:
> - serviceName: nginx-v1
> servicePort: 80
> - name: "be3"
> match:
> hosts:
> - d.testsssss.com
> paths:
> - "/*"
> backends:
> - serviceName: nginx-v1
> servicePort: 80
> ```
>
> 在 apisix/core/config_etcd.lua 351-356 中添加了调试日志记录
>
> ```
> 348 if not res then
> 349 return false, err
> 350 end
> 351 if res[1] and res[1].value and res[1].value.hosts then
> 352 local host = res[1].value.host or res[1].value.hosts[1]
> 353 if ngx.re.find(host, [[^[a-z]\.testsssss\.com$]], "jo") then
> 354 log.error("new data coming ", ngx.worker.id(), " pid ", ngx.worker.pid() , " host ", host)
> 355 end
> 356 end
> 357
> 358 local res_copy = res
> 359 -- waitdir will return [res] even for self.single_item = true
> 360 for _, res in ipairs(res_copy) do
> ```
>
> 工人人数 4
>
> 创建 APIsix 路由后的日志输出 kubectl create -f test.yaml
>
> 日志输出发现worker同步etcd数据不完整
>
> ```
> 2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host a.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host a.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 50#50: *203538 [lua] config_etcd.lua:354: sync_data(): new data coming nil pid 50 host a.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 44#44: *205783 [lua] config_etcd.lua:354: sync_data(): new data coming 0 pid 44 host a.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 47#47: *202984 [lua] config_etcd.lua:354: sync_data(): new data coming 3 pid 47 host a.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host b.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host b.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host c.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host c.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 46#46: *203557 [lua] config_etcd.lua:354: sync_data(): new data coming 2 pid 46 host d.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 45#45: *203467 [lua] config_etcd.lua:354: sync_data(): new data coming 1 pid 45 host d.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 44#44: *205783 [lua] config_etcd.lua:354: sync_data(): new data coming 0 pid 44 host b.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 47#47: *202984 [lua] config_etcd.lua:354: sync_data(): new data coming 3 pid 47 host b.testsssss.com, context: ngx.timer
> 2022/06/24 07:13:46 [error] 50#50: *203538 [lua] config_etcd.lua:354: sync_data(): new data coming nil pid 50 host b.testsssss.com, context: ngx.timer
> ```
>
> 创建的第三条和第四条路由不同步，存在于 ETCD 数据 中 有问题吗？或者我是否以错误的方式使用它，到目前为止我有两个可以重现的环境。
>
> ### 环境
> * APISIX 版本（运行`apisix version`）：：2.11.0
> * 操作系统（运行`uname -a`）：k8s v1.22.7
> * OpenResty / Nginx 版本（运行`openresty -V`或`nginx -V`）：2.11.0
> * etcd 版本，如果相关（运行`curl http://127.0.0.1:9090/v1/server_info`）：v3.5.4
> * APISIX 仪表板版本（如果相关）：
> * 插件运行器版本，针对与插件运行器相关的问题：
> * LuaRocks 版本，用于安装问题（运行`luarocks --version`）：
请问这个问题是如何解决的？我也遇到了这个问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Gerrylinux commented on issue #7322: The worker synchronizes etcd data. When more than 3 routes are created at the same time, the nginx worker cannot obtain the latest etcd data
Posted by GitBox <gi...@apache.org>.
Gerrylinux commented on issue #7322:
URL: https://github.com/apache/apisix/issues/7322#issuecomment-1214647044
你是否使用了 etcd gateway 用于链接 etcd ，我这里去掉了gateway 这层etcd转发后恢复。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Gerrylinux commented on issue #7322: The worker synchronizes etcd data. When more than 3 routes are created at the same time, the nginx worker cannot obtain the latest etcd data
Posted by GitBox <gi...@apache.org>.
Gerrylinux commented on issue #7322:
URL: https://github.com/apache/apisix/issues/7322#issuecomment-1214671143
我这边是使用了 etcd gateway 代理方式，是etcd 的功能。
`etcd gateway start --listen-addr=127.0.0.1:2379 --endpoints=https://10.10.10.1:2379,https://10.10.10.2:2379,https://10.10.10.3:2379 --trusted-ca-file=/tmp/ca.pem`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Gerrylinux closed issue #7322: The worker synchronizes etcd data. When more than 3 routes are created at the same time, the nginx worker cannot obtain the latest etcd data
Posted by GitBox <gi...@apache.org>.
Gerrylinux closed issue #7322: The worker synchronizes etcd data. When more than 3 routes are created at the same time, the nginx worker cannot obtain the latest etcd data
URL: https://github.com/apache/apisix/issues/7322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] whioue commented on issue #7322: The worker synchronizes etcd data. When more than 3 routes are created at the same time, the nginx worker cannot obtain the latest etcd data
Posted by GitBox <gi...@apache.org>.
whioue commented on issue #7322:
URL: https://github.com/apache/apisix/issues/7322#issuecomment-1214649284
> etcd gateway
请问这个配置是在哪里？我好像没有太明白，麻烦您能说得更清楚一些吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org