You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "yuqiquan (via GitHub)" <gi...@apache.org> on 2023/04/07 10:12:55 UTC
[GitHub] [apisix] yuqiquan opened a new issue, #9263: apisix etcd配置https watch监听机制异常
yuqiquan opened a new issue, #9263:
URL: https://github.com/apache/apisix/issues/9263
### Description
先将etcd集群在某一台apisix实例中配置一条路由，路由详情如下：
```
{
"uri": "*",
"name": "etcd数据库",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"host": "yuqiquan.made-in-china.com",
"upstream": {
"nodes": [
{
"host": "10.110.3.50",
"port": 2379,
"weight": 1
},
{
"host": "10.110.3.51",
"port": 2379,
"weight": 1
},
{
"host": "10.110.3.52",
"port": 2379,
"weight": 1
}
],
"timeout": {
"connect": 120,
"send": 120,
"read": 120
},
"type": "roundrobin",
"scheme": "http",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 60000000,
"requests": 10000,
"size": 320
}
},
"status": 1
}
```
然后配置证书
我能确保它http，https请求都是通的，正常的
之后我换了一台apisix实例机器，在这台实例apisix中，有关etcd配置如下：
```
etcd:
host:
- "https://yuqiquan.made-in-china.com:443"
prefix: /test114 # apisix configurations prefix
timeout: 30 # 30 seconds
#resync_delay: 5 # when sync failed and a rest is needed, resync after the configured seconds plus 50% random jitter
health_check_timeout: 10 # etcd retry the unhealthy nodes after the configured seconds
health_check_retry: 2 # etcd retry time that only affects the health check, default 2
user: fwd # root username for etcd
password: fwd # root password for etcd
tls:
cert: /usr/local/apisix/conf/server.crt
key: /usr/local/apisix/conf/server.key
verify: false
sni: yuqiquan.made-in-china.com
```
同时我将 ssl 的 ssl_trusted_certificate 注释掉
然后启动apisix ，发现启动以后 etcd里的数据 apisix能够获取到，但是如果我在dashboard中修改了路由信息，数据并不能同步到apisix，这里我就觉得是监听出了状况，查看源码，知道监听机制的超时时间是30s。结合日志 在这30秒内，发现修改数据监听不到。 相关etcd 访问日志如下：
而如果etcd host 配置成 http://yuqiquan.made-in-china.com:80 就是正常的
etcd访问日志是这样
我分析了下lua 源码，发现在执行监听的时候
config_etcd.lua 中的
会返回nil，timeout
https的会出现timeout的异常，而http的没有
想请教下这个问题该怎么解决
### Environment
- APISIX version (run `apisix version`): 2.15.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kingluo commented on issue #9263: bug: apisix etcd https watch timeout issue
Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9263:
URL: https://github.com/apache/apisix/issues/9263#issuecomment-1500791663
What's your deployment topo? and which APISIX do the logs come from?
Could you clarify more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yuqiquan commented on issue #9263: bug: apisix etcd https watch timeout issue
Posted by "yuqiquan (via GitHub)" <gi...@apache.org>.
yuqiquan commented on issue #9263:
URL: https://github.com/apache/apisix/issues/9263#issuecomment-1501135092
1. 我的安装是用 RPM 方式的
2. 主要涉及部署了两个apisix实例， 可以叫他 apisix1 ， apisix2
3. 在apisix1中创建一条路由，上游是一个etcd集群，然后保证http https 请求它都是正常的 (这个实例使用的etcd配置是这样的)
路由详情
```
{
"uri": "*",
"name": "etcd数据库",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"host": "www.etcd.com",
"upstream": {
"nodes": [
{
"host": "10.110.3.50",
"port": 2379,
"weight": 1
},
{
"host": "10.110.3.51",
"port": 2379,
"weight": 1
},
{
"host": "10.110.3.52",
"port": 2379,
"weight": 1
}
],
"timeout": {
"connect": 120,
"send": 120,
"read": 120
},
"type": "roundrobin",
"scheme": "http",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 60000000,
"requests": 10000,
"size": 320
}
},
"status": 1
}
```
4. 在apisix2中我的etcd配置做了一些改变，想通过访问apisix1的 www.etcd.com 路由 来配置etcd集群， 所以 apisix2 etcd配置部分是这样的
```
etcd:
host:
- "https://www.etcd.com:443"
prefix: /test114 # apisix configurations prefix
timeout: 30 # 30 seconds
#resync_delay: 5 # when sync failed and a rest is needed, resync after the configured seconds plus 50% random jitter
health_check_timeout: 10 # etcd retry the unhealthy nodes after the configured seconds
health_check_retry: 2 # etcd retry time that only affects the health check, default 2
user: fwd # root username for etcd
password: fwd # root password for etcd
tls:
#cert: /usr/local/apisix/conf/server.crt
#key: /usr/local/apisix/conf/server.key
verify: false
sni: yuqiquan.made-in-china.com
```
同时我将 ssl 的 ssl_trusted_certificate 注释掉
https://user-images.githubusercontent.com/44921350/230589159-de2970ca-8a0d-4037-a2a5-b4798a2522e3.png
5. 启动apisix2 ，发现日志是正常的，同时 apisix1 的access.log中出现访问/v3/.... 的一些路径的路由，但是30秒以后，apisix1中就会出现很多这样日志
https://user-images.githubusercontent.com/44921350/230592860-b8cfcfd0-96fd-4236-9058-8aa58093afec.png
6. 日志内容是通过我的apisix2 配置的etcd ，apisix2 底层代码会访问 apisix1的 https://www.etcd.com:443/v3/watch的路由产生的
7. 在apisix2 启动以后，我尝试改变apisix2的路由数据信息，发现路由数据无法监听获取到改变， 然后我尝试将apisix2的etcd配置改为 http://www.etcd.com:80 发现监听是正常的
扒了下源码 发现在执行监听的时候
config_etcd.lua 中的
https://user-images.githubusercontent.com/44921350/230590927-621b15d4-3e4c-4043-b9e0-4f26f499f8fc.png
配置成https时 ，会返回nil，timeout ，而http的没有，不知道为什么， 就是请求etcd的监听 https的老是监听不到
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kingluo commented on issue #9263: bug: apisix etcd https watch timeout issue
Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9263:
URL: https://github.com/apache/apisix/issues/9263#issuecomment-1505524557
@yuqiquan Yes, I confirm there is some issue to route the etcd traffic via APISIX: it does not forward the response chunks from upstream to downstream (etcd already responds watch result chunk). This issue exists for both http and https downstreams (you could double-check the http case).
In fact, if you use the base nginx functionality of APISIX, it works:
`config.yaml`
```yaml
nginx_config:
http_configuration_snippet: |
server {
listen 20000;
location / {
proxy_pass http://127.0.0.1:2379;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
```
I will try to find out the cause of the problem later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kingluo commented on issue #9263: bug: proxy etcd via apisix: watch timeout: no response
Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9263:
URL: https://github.com/apache/apisix/issues/9263#issuecomment-1506475658
@yuqiquan When `proxy_buffering` is `on` (by default), the downstream SSL response will be buffered. So you could not watch anything and timeout finally.
After testing, the solution is set `proxy_buferring` to `off`.
`config.yaml`
```yaml
nginx_config:
http_configuration_snippet: |
proxy_buffering off;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kingluo commented on issue #9263: bug: apisix etcd https watch timeout issue
Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9263:
URL: https://github.com/apache/apisix/issues/9263#issuecomment-1501709709
I'll try to reproduce the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yuqiquan commented on issue #9263: bug: proxy etcd via apisix: watch timeout: no response
Posted by "yuqiquan (via GitHub)" <gi...@apache.org>.
yuqiquan commented on issue #9263:
URL: https://github.com/apache/apisix/issues/9263#issuecomment-1506521906
太感谢了 ，我将用这个方法去测试一下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yuqiquan closed issue #9263: bug: proxy etcd via apisix: watch timeout: no response
Posted by "yuqiquan (via GitHub)" <gi...@apache.org>.
yuqiquan closed issue #9263: bug: proxy etcd via apisix: watch timeout: no response
URL: https://github.com/apache/apisix/issues/9263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org