You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/11/26 07:37:03 UTC
[GitHub] [apisix] yangxikun opened a new issue #5622: bug: APISIX occasional blocking
yangxikun opened a new issue #5622:
URL: https://github.com/apache/apisix/issues/5622
### Issue description
APISIX Pod occasional failed on readiness probe(GET /healthz). I wrote a program to do http health check and log netstat output.
My program error log:
```
2021/11/26 01:10:45 err: Get "http://localhost:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/11/26 01:10:45 2021-11-26 01:10:43.345216628 +0000 UTC m=+39831.465577142 Get Conn: localhost:8080
2021-11-26 01:10:43.345260592 +0000 UTC m=+39831.465621171 DNS Start: {Host:localhost}
2021-11-26 01:10:43.345368086 +0000 UTC m=+39831.465728613 DNS Info: {Addrs:[{IP:127.0.0.1 Zone:} {IP:::1 Zone:}] Err:<nil> Coalesced:false}
2021-11-26 01:10:43.34539942 +0000 UTC m=+39831.465759942 Connect Start: tcp 127.0.0.1:8080
2021-11-26 01:10:43.649157665 +0000 UTC m=+39831.769518188 Connect Start: tcp [::1]:8080
2021-11-26 01:10:43.649219174 +0000 UTC m=+39831.769579704 Connect Done: tcp [::1]:8080 dial tcp [::1]:8080: connect: cannot assign requested address
2021-11-26 01:10:45.345542547 +0000 UTC m=+39833.465903093 Connect Done: tcp 127.0.0.1:8080 dial tcp 127.0.0.1:8080: i/o timeout
```
AND the netstat show APISIX is not read network packet(recv-q > 0):
### Environment
- apisix version (cmd: `apisix version`): 2.10.0
- OS (cmd: `uname -a`): 4.14.105-1-tlinux3-0016 #1 SMP Fri Jul 31 16:16:47 CST 2020 x86_64 x86_64 x86_64 GNU/Linux
- OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`): nginx version: openresty/1.19.9.1
built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
built with OpenSSL 1.1.1k 25 Mar 2021 (running with OpenSSL 1.1.1l 24 Aug 2021)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.20 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.10 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/loc
al/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --with-cc='ccache gcc -fdiagnostics-color=always' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
- etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API):
- apisix-dashboard version, if have:
- the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
- luarocks version, if the issue is about installation (cmd: `luarocks --version`):
### Steps to reproduce
It's occasionally, I cannot provide reproduce steps.
### Actual result
APISIX blocking temporary. Cause some request fail.
### Error log
No any error log.
### Expected result
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #5622:
URL: https://github.com/apache/apisix/issues/5622#issuecomment-981229992
I see this error log: `cannot assign requested address`
Is this a problem with your environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yangxikun commented on issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
yangxikun commented on issue #5622:
URL: https://github.com/apache/apisix/issues/5622#issuecomment-979822857
@tokers 200 response
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yangxikun closed issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
yangxikun closed issue #5622:
URL: https://github.com/apache/apisix/issues/5622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #5622:
URL: https://github.com/apache/apisix/issues/5622#issuecomment-979817556
What's the expected behavior of the `/healthz` API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yangxikun commented on issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
yangxikun commented on issue #5622:
URL: https://github.com/apache/apisix/issues/5622#issuecomment-982408979
This maybe our host VM problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yangxikun edited a comment on issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
yangxikun edited a comment on issue #5622:
URL: https://github.com/apache/apisix/issues/5622#issuecomment-980094603
Not just health check API, all other api request is block. @tokers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #5622:
URL: https://github.com/apache/apisix/issues/5622#issuecomment-979828870
> @tokers 200 response
I'm not sure how did you implement the health check API, and which logics inside there, so I cannot determine the reason that why this API will time out. Please provide more details about the topology, the chain of the health check request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] yangxikun commented on issue #5622: bug: APISIX occasional blocking
Posted by GitBox <gi...@apache.org>.
yangxikun commented on issue #5622:
URL: https://github.com/apache/apisix/issues/5622#issuecomment-980094603
Not just health check API, all other api request is block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org