You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "math-chao (via GitHub)" <gi...@apache.org> on 2023/05/11 07:18:52 UTC
[GitHub] [apisix] math-chao opened a new issue, #9464: help request: Program deadlock, cpu usage reaches 100%
math-chao opened a new issue, #9464:
URL: https://github.com/apache/apisix/issues/9464
### Description
version: apisix:2.15.0-centos in k8s
run for 13 days, and Program deadlock,
error log
prometheus.lua:920: log_error(): Error while setting 'node_info{hostname=""}' to '1': 'no memory', client: xxx.xxxx.xxxx, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "xxxx"
prometheus.lua:920: log_error(): Error while setting 'etcd_modify_indexes{key="x_etcd_index"}' to '8553921': 'no memory', client: xxx.xxxx.xxx, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "xxxx"
debug by strace, found a lot of futex calls
futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
debug by perf
Samples: 31K of event 'cycles', Event count (approx.): 18419237386
Children Self Command Shared Object Symbol
+ 87.95% 0.00% openresty [unknown] [k] 0x0000000000000080
+ 84.44% 83.62% openresty nginx [.] ngx_shmtx_lock
+ 4.85% 0.10% openresty [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 4.57% 0.03% openresty [kernel.kallsyms] [k] do_syscall_64
+ 4.36% 0.07% openresty [kernel.kallsyms] [k] __x64_sys_futex
+ 4.30% 0.08% openresty [kernel.kallsyms] [k] do_futex
pstack results are as follows
#0 0x0000000000449bf3 in ngx_shmtx_lock (mtx=0x7fc9d4ded068) at src/core/ngx_shmtx.c:78
#1 0x0000000000572caf in ngx_meta_lua_ffi_shdict_get (zone=0x1281138, key=0x7fc9d1d69080 "__ngx_prom__delete_count", key_len=24, value_type=0x7fc9de85f828, str_value_buf=0x7fc9de844380, str_value_len=0x7fc9de875ef8, num_value=0x7fc9de85f868, user_flags=0x7fc9de85f848, get_stale=0, is_stale=0x7fc9de8636f8, err=0x7fc9de8723a8) at /tmp/tmp.YN7822nAJt/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/meta/ngx_meta_lua_shdict.c:1631
#2 0x00007fca19caa8d7 in ?? ()
#3 0x00007fc9de85f868 in ?? ()
#4 0x00007fc9de85f848 in ?? ()
#5 0x0000000000000000 in ?? ()
What's the problem？help please
### Environment
- APISIX version (run `apisix version`): 2.15.0-centos
- Operating system (run `uname -a`): Linux 5.10.131-1.el8.x86_64 #1 SMP Sun Jul 17 11:20:32 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Program deadlock, cpu usage reaches 100% [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9464:
URL: https://github.com/apache/apisix/issues/9464#issuecomment-2016433237
Due to lack of the reporter's response this issue has been labeled with "no response". It will be close in 3 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Program deadlock, cpu usage reaches 100% [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on issue #9464:
URL: https://github.com/apache/apisix/issues/9464#issuecomment-1846492920
@math-chao do you have any updates here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Program deadlock, cpu usage reaches 100% [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9464:
URL: https://github.com/apache/apisix/issues/9464#issuecomment-2022363582
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Aaron199 commented on issue #9464: help request: Program deadlock, cpu usage reaches 100%
Posted by "Aaron199 (via GitHub)" <gi...@apache.org>.
Aaron199 commented on issue #9464:
URL: https://github.com/apache/apisix/issues/9464#issuecomment-1609230978
I have the same problem, and I have found some clues that I hope can help solve the issue.
<img width="1360" alt="image" src="https://github.com/apache/apisix/assets/35388612/d79c9da0-3110-4e84-8948-a963b987f642">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] fleashiro commented on issue #9464: help request: Program deadlock, cpu usage reaches 100%
Posted by "fleashiro (via GitHub)" <gi...@apache.org>.
fleashiro commented on issue #9464:
URL: https://github.com/apache/apisix/issues/9464#issuecomment-1543746468
I think you need to check the prometheus shared dict size first. Check metrics`apisix_shared_dict_capacity_bytes` and `apisix_shared_dict_free_space_bytes`. If the memory ran out, increase the size in your config file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Program deadlock, cpu usage reaches 100% [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9464: help request: Program deadlock, cpu usage reaches 100%
URL: https://github.com/apache/apisix/issues/9464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org