You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/05/25 06:27:25 UTC
[GitHub] [apisix] heming79 opened a new issue, #7125: bug: apisix2.9 one upstream fail all request return 502
heming79 opened a new issue, #7125:
URL: https://github.com/apache/apisix/issues/7125
### Current Behavior
[root@yz-mbg-018203 ~]# curl -voax service-bmlt.wanzhuanmohe.cn -x 192.168.7.204:80
* About to connect() to proxy 192.168.7.204 port 80 (#0)
* Trying 192.168.7.204...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.7.204 (192.168.7.204) port 80 (#0)
> GET HTTP://service-bmlt.wanzhuanmohe.cn/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: service-bmlt.wanzhuanmohe.cn
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 502 Bad Gateway
< Date: Wed, 25 May 2022 04:37:09 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 154
< Connection: keep-alive
< Server: APISIX/2.9
< X-APISIX-Upstream-Status: 502 :
<
{ [data not shown]
100 154 100 154 0 0 15837 0 --:--:-- --:--:-- --:--:-- 17111
* Connection #0 to host 192.168.7.204 left intact
去掉问题节点 马上就恢复200状态
[root@yz-mbg-018203 ~]# curl -voax service-bmlt.wanzhuanmohe.cn -x 192.168.7.204:80
* About to connect() to proxy 192.168.7.204 port 80 (#0)
* Trying 192.168.7.204...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.7.204 (192.168.7.204) port 80 (#0)
> GET HTTP://service-bmlt.wanzhuanmohe.cn/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: service-bmlt.wanzhuanmohe.cn
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200
< Content-Type: application/json;charset=utf-8
< Content-Length: 42
< Connection: keep-alive
< Date: Wed, 25 May 2022 04:41:19 GMT
< Server: APISIX/2.9
<
{ [data not shown]
100 42 100 42 0 0 8552 0 --:--:-- --:--:-- --:--:-- 10500
### Expected Behavior
只要有一个可用的upstream node ，就应该 返回200 状态 。
nginx就可以 proxy_next_upstream http_404 http_500 http_502 http_503 http_504 error timeout invalid_header;.
应该参考 nginx的 /nstatus 方便运维快速查看、监控所有upstream后端node的 健康状态 。
另外如果 discovery 能支持etcd 要是能像traefik那样 获取k8s的pod地址:端口号 就完美了 。
![Uploading image.png…]()
### Error Logs
我上午起过两个apisix 出现etcd 连接600多个导致 etcd 不可用的情况 。
/usr/local/apisix/logs # tail -f error.log
2022/05/25 02:27:30 [error] 63#63: *304 [lua] config_etcd.lua:563: no healthy etcd endpoint available, next retry after 2s, context: ngx.timer
2022/05/25 02:27:30 [error] 57#57: *239 [lua] config_etcd.lua:563: no healthy etcd endpoint available, next retry after 2s, context: ngx.timer
2022/05/25 02:27:31 [error] 50#50: *49 [lua] config_etcd.lua:563: no healthy etcd endpoint available, next retry after 2s, context: ngx.timer
2022/05/25 02:27:31 [error] 44#44: *77 [lua] config_etcd.lua:563: no healthy etcd endpoint available, next retry after 4s, context: ngx.timer
不知道什么原因 日志一直在切割归档，而不是zhengd整点才切割归档。
![Uploading image.png…]()
### Steps to Reproduce
1、run apisix docker ， dashboard docker
2、create upstream
3、create route
4、curl -voax service-bmlt.wanzhuanmohe.cn -x 192.168.7.204:80 OK
5、add upstream node （no availabe ）
6、curl -voax service-bmlt.wanzhuanmohe.cn -x 192.168.7.204:80 502
### Environment
- APISIX version (run `apisix version`):2.9
- Operating system (run `uname -a`):centos7.9
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):1.19.3.1
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):3.4.15
- APISIX Dashboard version, if relevant:2.8.0
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):3.7.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1282386496
> think there may some difference between web browser and pure http request? you can see that the browser return 502 when it request as http1.1
I don't have a better idea, you can grab the network packet sent by the browser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1281823747
> you can visit [http://design.ux.mojerro.wang](http://design.ux.mojerro.wang/) and have a look at 502 page
my route
```
curl http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
{
"uri": "/hello",
"upstream": {
"type": "roundrobin",
"nodes": {
"design.ux.mojerro.wang:80": 1
}
}
}'
```
and test
```
curl --http2-prior-knowledge "http://127.0.0.1:9080/hello" -vso /dev/null
* Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /hello]
* h2h3 [:scheme: http]
* h2h3 [:authority: 127.0.0.1:9080]
* h2h3 [user-agent: curl/7.84.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x2092f10)
> GET /hello HTTP/2
> Host: 127.0.0.1:9080
> user-agent: curl/7.84.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 000
< date: Tue, 18 Oct 2022 05:10:47 GMT
< server: APISIX/2.99.0
<
{ [57 bytes data]
* Connection #0 to host 127.0.0.1 left intact
```
I think this means that APISIX has successfully proxied the http2 request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] gxthrj commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1136909036
I hope you can provide the error log, when APISIX returns 502.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1280563691
> I reproduce it again, if I enable http2, then it returns 502 with no error log as blow.
> If I close it and recreate the docker container, it runs well.
It looks like you're deploying with docker, I'm wondering how you turned http2 on or off? Modify the config file and rebuild the APISIX docker?
> And I find one more thing, even with http2 = false, web can also use h2 proto
This is just a web page that shows that it uses the http2 protocol, it does not mean that APISIX actually handles the http2 protocol. In general, we focus on the `/usr/loca/apisix/conf/nginx.conf`.
if you enable http2 in `config.yaml` like:
```yaml
apisix:
node_listen:
- port: 9080
enable_http2: true
```
the `/usr/loca/apisix/conf/nginx.conf` mabe have `http2` like:
```nginx
server {
listen 0.0.0.0:9080 default_server http2 reuseport;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1281842655
@tzssangglass I think there may some difference between web browser and pure http request? you can see that the browser return 502 when it request as http1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] heming79 commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
heming79 commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1138072653
sometime still have 502 :
192.168.7.203 - - [26/May/2022:01:52:51 +0000] service-bmlt.wanzhuanmohe.cn "GET HTTP://service-bmlt.wanzhuanmohe.cn/ HTTP/1.1" 502 154 0.002 "-" "curl/7.29.0" 192.168.6.91:20730 : 502 : 0.001 : "http://service-bmlt.wanzhuanmohe.cn"
[root@yz-mbg-018204 logs]# tail -n 200 2022-05-26_02-00-00__error.log |grep "pick server"
2022/05/26 01:52:48 [error] 63#63: *21976125 [lua] balancer.lua:327: run(): failed to pick server: failed to find valid upstream server, all upstream servers tried while connecting to upstream, client: 192.168.7.203, server: _, request: "GET HTTP://service-bmlt.wanzhuanmohe.cn/ HTTP/1.1", upstream: "http://192.168.6.91:20730/", host: "service-bmlt.wanzhuanmohe.cn"
2022/05/26 01:52:50 [error] 51#51: *21976772 [lua] balancer.lua:327: run(): failed to pick server: failed to find valid upstream server, all upstream servers tried while connecting to upstream, client: 192.168.7.203, server: _, request: "GET HTTP://service-bmlt.wanzhuanmohe.cn/ HTTP/1.1", upstream: "http://192.168.6.91:20730/", host: "service-bmlt.wanzhuanmohe.cn"
2022/05/26 01:52:51 [error] 46#46: *21976925 [lua] balancer.lua:327: run(): failed to pick server: failed to find valid upstream server, all upstream servers tried while connecting to upstream, client: 192.168.7.203, server: _, request: "GET HTTP://service-bmlt.wanzhuanmohe.cn/ HTTP/1.1", upstream: "http://192.168.6.91:20730/", host: "service-bmlt.wanzhuanmohe.cn"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1277700909
我也遇到了这个问题，是线上环境，突然出现好几个接口502了，都是不常用的或者新建的接口。而且刚开始都是好的，直接访问端口也是畅通的。
现在我看access log和error log都很正常，返回502的接口好像没有单独的日志
@heming79 大佬方便到时候远程看看吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1279729045
Can you provide the reproduction steps? So that I can reproduce it locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1281214111
after you enable http2 in `./config/apisix_conf.yaml`, can you show the `/usr/loca/apisix/conf/nginx.conf` in docker, like:
```
server {
listen 0.0.0.0:9080
```
and if 502 appears, what are the error logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1279701327
@heming79 老哥，你开启http2了吗，我这边这两天研究了一下，还重新部署了一遍，发现开启http2之后，所有1.1的请求都直接502且无error log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1281790522
you can visit http://design.ux.mojerro.wang/ and have a look at 502 page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] moonming commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
moonming commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1138069966
> > > If your upstream does not have port 80, this will cause apisix to determine that all your upstream services are unhealthy, and then use the default nodes (all configured nodes) for back-to-origin
> > > It looks like apisix didn't kick out any unhealthy nodes.
> >
> >
> > In short, is this a bug in Apache APISIX?
>
> I think it's not. But if the dashboard forces `port` to be configured, that's a problem, I'll reconfirm this.
I am confused. In any case, one upstream failure cannot affect other upstreams.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1280206704
@tzssangglass I reproduce it again, if I enable http2, then it returns 502 with no error log as blow.
<img width="861" alt="image" src="https://user-images.githubusercontent.com/15071824/196078796-0efa41ee-6aa3-45f3-b790-d7c462919ca4.png">
<img width="421" alt="image" src="https://user-images.githubusercontent.com/15071824/196078950-dfb3df78-01b3-4817-a92a-15c7c0489c95.png">
If I close it and recreate the docker container, it runs well.
<img width="1141" alt="image" src="https://user-images.githubusercontent.com/15071824/196079080-8407e7ef-2845-4f0f-8a3c-89b8b8bf663a.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1277702128
版本信息
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] heming79 commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
heming79 commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1138145336
可能还是因为老版本dashboard 导致后端检测用了默认port:80 ， 当时其他节点都不可用只有192.168.6.91:20730 主动检测可用 ，实际被动检测502 ，所以就没有尝试别的节点 （主动检测异常的节点此时是不允许尝试的 ，/v1/healthycheck 目前不稳定所以我看不到后端健康检测实际情况 无法做出明确判断）。 应该不是bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1280188826
> @tzssangglass sorry for the late reply. Here are the steps.
>
> 1. enable `enable_http2: true` in `node_listen`
> 2. start the apisix
> 3. all requests with h1.1 will return 502 directly. You can deploy [penpot](https://help.penpot.app/technical-guide/getting-started/) and access it through apisix which will get this error.
I cannot be reproduced.
my step:
`config.yaml`:
```
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
admin_key:
- name: admin
key: edd1c9f034335f136f87ad84b625c8f1 # using fixed API token has security risk, please update it when you deploy to production environment
role: admin
node_listen:
- port: 9080
enable_http2: true
```
route config:
```
curl http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
{
"uri": "/get",
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org:80": 1
}
}
}'
```
test:
```
curl http://127.0.0.1:9080/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "127.0.0.1",
"User-Agent": "curl/7.79.1",
"X-Amzn-Trace-Id": "Rxxxxxx",
"X-Forwarded-Host": "127.0.0.1"
},
"origin": "xxxxxx",
"url": "http://127.0.0.1/get"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1280582719
here is the `docker-compose.yml`
```yml
apisix:
image: apache/apisix:2.15.0-alpine
restart: unless-stopped
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
- TZ=${TIMEZONE}
volumes:
- ./apisix_log:/usr/local/apisix/logs
- ./config/apisix_conf.yaml:/usr/local/apisix/conf/config.yaml:ro
depends_on:
- etcd
##network_mode: host
ports:
- "80:80/tcp"
- "39091:9091/tcp"
- "443:443/tcp"
- "39180:9180/tcp"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix2.9 one upstream node fail all request return 502 [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1746549960
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix2.9 one upstream node fail all request return 502 [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7125: bug: apisix2.9 one upstream node fail all request return 502
URL: https://github.com/apache/apisix/issues/7125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1278554757
> > 现在我看access log和error log都很正常，返回502的接口好像没有单独的日志
> > @heming79 大佬方便到时候远程看看吗
>
> Can you reproduce it?
yes, it still return 502 on my server now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1282530635
> > think there may some difference between web browser and pure http request? you can see that the browser return 502 when it request as http1.1
>
> I don't have a better idea, you can grab the network packet sent by the browser
OK, I will have a try.
Anyway, thanks for your reply~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] moonming commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
moonming commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1138058366
> If your upstream does not have port 80, this will cause apisix to determine that all your upstream services are unhealthy, and then use the default nodes (all configured nodes) for back-to-origin
> It looks like apisix didn't kick out any unhealthy nodes.
In short, is this a bug in Apache APISIX?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] gxthrj commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1136907985
I tried with APISIX 2.9 and etcd 3.5 and found no such problem.
Use `httpbin.org` as upstream.
```
# curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
> {
> "uri": "/ip",
> "upstream": {
> "type": "roundrobin",
> "nodes": {
> "httpbin.org": 1
> }
> }
> }'
```
and then I add `127.0.0.1:80` which is not exist.
```
# curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
> {
> "uri": "/ip",
> "upstream": {
> "type": "roundrobin",
> "nodes": {
> "httpbin.org": 1,
> "127.0.0.1:80": 1
> }
> }
> }'
```
The result is OK.
```
# curl http://127.0.0.1:9080/ip
{
"origin": "10.100.0.1"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1280580481
> It looks like you're deploying with docker, I'm wondering how you turned http2 on or off? Modify the config file and rebuild the APISIX docker?
My command shows below, and my env file is in `./config/apisix_config.yml`
```sh
# start apisix
docker compose -p apisix up -d
# down apisix
docker compose -p apisix down
```
Thanks for your explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1277783911
> 现在我看access log和error log都很正常，返回502的接口好像没有单独的日志
> @heming79 大佬方便到时候远程看看吗
Can you reproduce it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #7125: bug: apisix2.9 one upstream fail all request return 502
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1136824441
> 1、run apisix docker ， dashboard docker
> 2、create upstream
> 3、create route
> 4、curl -voax service-bmlt.wanzhuanmohe.cn -x 192.168.7.204:80 OK
> 5、add upstream node （no availabe ）
> 6、curl -voax service-bmlt.wanzhuanmohe.cn -x 192.168.7.204:80 502
can you provide the detail information about `2`, `3`, `5`? many thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1138044082
I noticed that you actually have two problems here:
1. Unhealthy nodes are not removed
2. From the log, it seems that there is no retry after the upstream returns 502
Regarding the first question, because you have configured checks.active.port to 80, healthcheck will use this port first when performing health checks.
If your upstream does not have port 80, this will cause apisix to determine that all your upstream services are unhealthy, and then use the default nodes (all configured nodes) for back-to-origin
It looks like apisix didn't kick out any unhealthy nodes.
If you look at the error_log, you should see something like this: `all upstream nodes is unhealthy, use default`.
So you should remove the `port:80` configuration. In addition, it is best to add 502 status code to the `checks.unhealthy.http_statuses` configuration
The second problem seems to be unexpectedly exiting in the process of retrying. I noticed that you configured `retry_timeout: 2`, if the network fluctuates and the first node is connected for more than 2s, then the next node will not have a chance to be retried.
Refer to this code: https://github.com/apache/apisix/blob/master/apisix/balancer.lua#L346-L350. If you can provide me with an errorlog, I will be able to confirm if this is the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1138060703
> > If your upstream does not have port 80, this will cause apisix to determine that all your upstream services are unhealthy, and then use the default nodes (all configured nodes) for back-to-origin
> > It looks like apisix didn't kick out any unhealthy nodes.
>
> In short, is this a bug in Apache APISIX?
I thinke it's not. But if the dashboard forces `port` to be configured, that's a problem, I'll reconfirm this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1281788887
@tzssangglass if 502 appears, there is nothing in `error.logs` and docker logs.
`/usr/loca/apisix/conf/nginx.conf`
```conf
server {
listen 0.0.0.0:9080 default_server http2 reuseport;
listen 0.0.0.0:9443 ssl default_server http2 reuseport;
server_name _;
ssl_certificate cert/ssl_PLACE_HOLDER.crt;
ssl_certificate_key cert/ssl_PLACE_HOLDER.key;
ssl_session_cache shared:SSL:20m;
ssl_session_timeout 10m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers on;
ssl_session_tickets off;
# http server configuration snippet starts
# http server configuration snippet ends
location = /apisix/nginx_status {
allow 127.0.0.0/24;
deny all;
access_log off;
stub_status;
}
ssl_certificate_by_lua_block {
apisix.http_ssl_phase()
}
proxy_ssl_name $upstream_host;
proxy_ssl_server_name on;
location / {
set $upstream_mirror_uri '';
set $upstream_upgrade '';
set $upstream_connection '';
set $upstream_scheme 'http';
set $upstream_host $http_host;
set $upstream_uri '';
set $ctx_ref '';
set $from_error_page '';
# http server location configuration snippet starts
# http server location configuration snippet ends
access_by_lua_block {
apisix.http_access_phase()
}
proxy_http_version 1.1;
proxy_set_header Host $upstream_host;
proxy_set_header Upgrade $upstream_upgrade;
proxy_set_header Connection $upstream_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass_header Date;
### the following x-forwarded-* headers is to send to upstream server
set $var_x_forwarded_for $remote_addr;
set $var_x_forwarded_proto $scheme;
set $var_x_forwarded_host $host;
set $var_x_forwarded_port $server_port;
if ($http_x_forwarded_for != "") {
set $var_x_forwarded_for "${http_x_forwarded_for}, ${realip_remote_addr}";
}
if ($http_x_forwarded_host != "") {
set $var_x_forwarded_host $http_x_forwarded_host;
}
if ($http_x_forwarded_port != "") {
set $var_x_forwarded_port $http_x_forwarded_port;
}
proxy_set_header X-Forwarded-For $var_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $var_x_forwarded_proto;
proxy_set_header X-Forwarded-Host $var_x_forwarded_host;
proxy_set_header X-Forwarded-Port $var_x_forwarded_port;
### the following configuration is to cache response content from upstream server
set $upstream_cache_zone off;
set $upstream_cache_key '';
set $upstream_cache_bypass '';
set $upstream_no_cache '';
proxy_cache $upstream_cache_zone;
proxy_cache_valid any 10s;
proxy_cache_min_uses 1;
proxy_cache_methods GET HEAD POST;
proxy_cache_lock_timeout 5s;
proxy_cache_use_stale off;
proxy_cache_key $upstream_cache_key;
proxy_no_cache $upstream_no_cache;
proxy_cache_bypass $upstream_cache_bypass;
proxy_pass $upstream_scheme://apisix_backend$upstream_uri;
mirror /proxy_mirror;
header_filter_by_lua_block {
apisix.http_header_filter_phase()
}
body_filter_by_lua_block {
apisix.http_body_filter_phase()
}
log_by_lua_block {
apisix.http_log_phase()
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1280158999
> Can you provide the reproduction steps? So that I can reproduce it locally.
@tzssangglass sorry for the late reply. Here are the steps.
1. enable `enable_http2: true` in `node_listen`
2. start the apisix
3. all requests with h1.1 will return 502 directly. You can deploy [penpot](https://help.penpot.app/technical-guide/getting-started/) and access it through apisix which will get this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] mojerro commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
mojerro commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1280210415
for common errors with cannot access upstream services, apisix usually return like this
<img width="340" alt="image" src="https://user-images.githubusercontent.com/15071824/196080300-7e0611f8-72a3-4b52-88e3-766674146124.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] heming79 commented on issue #7125: bug: apisix2.9 one upstream node fail all request return 502
Posted by GitBox <gi...@apache.org>.
heming79 commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1137021855
升级到 2.13.1之后 好很多了 。 偶尔还是会出现 502 不尝试next upstream的情况 。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix2.9 one upstream node fail all request return 502 [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7125:
URL: https://github.com/apache/apisix/issues/7125#issuecomment-1770487183
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org