You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/07/27 13:34:35 UTC
[GitHub] [apisix] ryan4yin opened a new issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
ryan4yin opened a new issue #4685:
URL: https://github.com/apache/apisix/issues/4685
### Issue description
when I set the upstreams's `retries=0` to disable the retry mechanism, the access_log shows that apisix retries many times.
access_log by example:
```
{ "time_iso8601": "2021-07-26T20:33:30+00:00", "request_method": "GET", "uri": "/xxx", "server_protocol": "HTTP/1.1", "status": "502", "request_time": "5.170", "body_bytes_sent": "154", "scheme": "http", "upstream_addr": "10.100.156.166:8080, 10.100.156.166:8080", "upstream_status": "502, 502", "upstream_response_time": "3.024, 2.148", "upstream_connect_time": "0.000, 1.124", "upstream_header_time": "-, -"}
{ "time_iso8601": "2021-07-26T20:33:30+00:00", "request_method": "GET", "uri": "xxx", "server_protocol": "HTTP/1.1", "status": "502", "request_time": "8.223", "body_bytes_sent": "154", "scheme": "http", "upstream_addr": "10.100.156.166:8080, 10.100.156.166:8080", "upstream_status": "502, 502", "upstream_response_time": "6.076, 2.148", "upstream_connect_time": "0.000, 1.124", "upstream_header_time": "-, -"}
{ "time_iso8601": "2021-07-26T20:33:30+00:00", "request_method": "GET", "uri": "/xxx", "server_protocol": "HTTP/1.1", "status": "502", "request_time": "2.596", "body_bytes_sent": "154", "scheme": "http", "upstream_addr": "10.100.156.166:8080, 10.100.156.166:8080", "upstream_status": "502, 502", "upstream_response_time": "0.444, 2.152", "upstream_connect_time": "0.000, 1.128", "upstream_header_time": "-, -"}
```
### Environment
use official docker image: `apache/apisix:2.5-alpine`
* apisix version (cmd: `apisix version`): 2.5
* OS (cmd: `uname -a`): Linux apisix-6f65f5bc9-km66f 4.14.209-160.335.amzn2.x86_64 #1 SMP Wed Dec 2 23:31:46 UTC 2020 x86_64 Linux
* OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`):
```
nginx version: openresty/1.19.3.1
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.9 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/pcre/l
ib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib' --with-pcre --with-compat --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-ipv6 --with-mail --with-mail_ssl_module --with-md5-asm --with-pcre-jit --with-sha1-asm --with-stream --with-stream_ssl_module --with-threads --with-stream --with-stream_ssl_preread_module
```
* etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API):
* luarocks version, if the issue is about installation (cmd: `luarocks --version`): v3.4.14
### Minimal test code / Steps to reproduce the issue
I failed to reproduce this problem with a minimal test env, it occurs in production environment WHEN the concurrent connections are very high - maybe more than 30000.
the apisix config in our production env(I mocked some id/key):
```shell
curl -X PUT http://localhost:9180/apisix/admin/upstreams/xxx -H 'X-API-KEY: xxx' --data-raw \
'{
"nodes": [
{
"port": 8080,
"priority": 0,
"weight": 1,
"host": "172.11.160.26"
}
],
"retries": 0,
"scheme": "http",
"pass_host": "pass",
"hash_on": "vars",
"type": "roundrobin"
}'
curl -X PUT http://localhost:9180/apisix/admin/services/xxx -H 'X-API-KEY: xxx' --data-raw \
'{
"plugins": {
"limit-conn": {
"default_conn_delay": 0.1,
"burst": 0,
"rejected_code": 529,
"conn": 32000,
"key": "server_addr"
},
"limit-req": {
"key": "server_addr",
"rate": 25000,
"burst": 50000,
"rejected_code": 519
},
"prometheus": {}
},
"upstream_id": "xxx",
}'
curl -X PUT http://localhost:9180/apisix/admin/routes/xxx -H 'X-API-KEY: xxx' --data-raw \
'{
"host": "eks-test",
"uri": "/*",
"upstream_id": "xxx"
}'
```
### What's the actual result? (including assertion message & call stack if applicable)
apisix retryed some requests.
### What's the expected result?
apisix should not retry any requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-887943412
Is `172.11.160.26` the host address in production? It's not matched with the one in access.log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-887966672
> > `access.log` is the log of client requests. For the retry, you should show the `error.log`.
>
> @tzssangglass will retry print log in error.log? any example logs?
take a look at this two test cases:https://github.com/apache/apisix/blob/1ab12ac2b0173f511e2427e3896df0154fdd34fd/t/node/upstream-retries.t#L170-L211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin commented on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin commented on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-887958846
> `access.log` is the log of client requests. For the retry, you should show the `error.log`.
@tzssangglass will retry print log in error.log? any example logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-887938622
`access.log` is the log of client requests. For the retry, you should show the `error.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs, and this error has no effect on upstream server.
The 502 is NOT retured by upstream, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 19:46:36 [error] 73#73: *5812952 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
Some cases that may cause this error:
- upstream has closed the connection, but nginx DO NOT know it yet, nginx still send request using this connection.
- upstream server has dead, but the linux kernel are still listening the port.
kill the upstream server may cause this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin commented on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin commented on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream,
the 502 indicates nginx failed to read data from upstream, we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs,
the 502 indicates nginx failed to read data from upstream, we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs,
the 502 is NOT retured by upstream, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs,
the 502 is NOT retured by upstream, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 19:46:36 [error] 73#73: *5812952 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
Some cases that may cause this error:
- upstream has closed the connection, but nginx DO NOT know it yet, nginx still send request using this connection.
- upstream server has dead, but the linux kernel are still listening the port.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have reproduce this error using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs, and this error has no effect on upstream server.
The 502 is NOT retured by upstream, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 19:46:36 [error] 73#73: *5812952 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
Some cases that may cause this error:
- upstream has closed the connection, but nginx DO NOT know it yet, nginx still send request using this connection.
- upstream server has dead, but the linux kernel are still listening the port.
kill the upstream server may cause this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs,
the 502 is NOT retured by upstrea, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs,
the 502 is NOT retured by upstream, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 19:46:36 [error] 73#73: *5812952 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
Some cases that may cause this error:
- upstream has closed the connection, but nginx DO NOT know it yet, nginx still send request using this connection.
- upstream server has dead, but the linux kernel are still listening the port.
kill the upstream server may cause this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin closed issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin closed issue #4685:
URL: https://github.com/apache/apisix/issues/4685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs,
the 502 is NOT retured by upstream, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 19:46:36 [error] 73#73: *5812952 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin commented on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin commented on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-887958292
> Is `172.11.160.26` the host address in production? It's not matched with the one in access.log.
sorry I confirm the address, it's matched `10.100.156.166` in access.log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] ryan4yin edited a comment on issue #4685: bug: apisix retries the requests when set upstream's `retries=0`
Posted by GitBox <gi...@apache.org>.
ryan4yin edited a comment on issue #4685:
URL: https://github.com/apache/apisix/issues/4685#issuecomment-888164938
we have repruduce this using nginx 1.19.3, it's the behavior of nginx upstream & keepalive configs,
the 502 is NOT retured by upstream, it just indicates that nginx failed to read data from upstream, it we found the corresponding log in `error.log`(I mocked some infos):
```
2021/07/27 13:30:30 [error] 59#59: *34290 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:33:23 [error] 70#70: *74921 upstream prematurely closed connection while reading response header from upstream, client: 172.11.193.70, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:36:55 [error] 62#62: *130238 upstream prematurely closed connection while reading response header from upstream, client: 172.11.187.255, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 13:42:36 [error] 61#61: *221268 upstream prematurely closed connection while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
2021/07/27 19:46:36 [error] 73#73: *5812952 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.11.145.126, server: _, request: "GET /eks-test HTTP/1.1", upstream: "http://10.100.154.110:8080/eks-test", host: "test.xxx"
```
Some cases that may cause this error:
- upstream has closed the connection, but nginx DO NOT know it yet, nginx still send request using this connection.
- upstream server has dead, but the linux kernel are still listening the port.
kill the upstream server will cause this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org