You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2020/12/24 10:00:18 UTC
[GitHub] [incubator-brpc] weixinwei opened a new issue #1311: glibc malloc deadlock
weixinwei opened a new issue #1311:
URL: https://github.com/apache/incubator-brpc/issues/1311
我们在业务中使用了brpc，现象是一些brpc bthread hung在了glibc的malloc和init_free。导致brpc所有bthread都hung住（业务原因在等其他bthread的资源）
不确定是否和brpc有关系。
**Versions (各种版本)**
OS: CentOS Linux release 8.1.1911 (Core) 4.18.0-147.el8.x86_64
Compiler: gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)
-DCMAKE_BUILD_TYPE="Debug" -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" -DCMAKE_CXX_FLAGS="-O0 -g3 -gdwarf-4"
brpc: brpc-0.9.8_rc01-1.el8.x86_64（tag 0.9.8-rc01源码build的rpm包）
protobuf: protobuf-3.5.0-7.el8.x86_64
glibc: glibc-2.28-101.el8.x86_64
**Additional context/screenshots (更多上下文/截图)**
Thread 83 (Thread 0x7f2d597fa700 (LWP 3633599)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef01e10 in _int_free (av=0x7f2d90000020, p=0x7f2d90204ad0, have_lock=<optimized out>) at malloc.c:4314
#2 0x0000559de3562314 in std::default_delete<brpc::Controller>::operator() (this=0x7f2e306f79a0, __ptr=0x7f2d90204ae0) at /usr/include/c++/8/bits/unique_ptr.h:81
#3 0x0000559de3561daf in std::unique_ptr<brpc::Controller, std::default_delete<brpc::Controller> >::~unique_ptr (this=0x7f2e306f79a0, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:269
Thread 79 (Thread 0x7f2d5b7fe700 (LWP 3633595)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef042d3 in __malloc_fork_lock_parent () at arena.c:156
#2 0x00007f2e6ef473ea in __libc_fork () at ../sysdeps/nptl/fork.c:73
#3 0x00007f2e6eef1984 in _IO_new_proc_open (fp=fp@entry=0x7f2d4c06ce40, command=command@entry=0x7f2e7e1760ae "uname -ap", mode=<optimized out>, mode@entry=0x7f2e7e343fae "r") at iopopen.c:122
#4 0x00007f2e6eef1c2c in _IO_new_popen (command=0x7f2e7e1760ae "uname -ap", mode=mode@entry=0x7f2e7e343fae "r") at iopopen.c:203
#5 0x00007f2e7df4ef84 in butil::read_command_output_through_popen (os=..., cmd=<optimized out>) at /usr/src/debug/brpc-0.9.8_rc01-1.el8.x86_64/src/butil/popen.cpp:159
#6 0x00007f2e7df5dcf7 in bvar::ReadVersion::ReadVersion (this=0x7f2d4c05fac0) at /usr/src/debug/brpc-0.9.8_rc01-1.el8.x86_64/src/bvar/default_variables.cpp:608
#7 0x00007f2e7df5dfac in butil::GetLeakySingleton<bvar::ReadVersion>::create_leaky_singleton () at /usr/src/debug/brpc-0.9.8_rc01-1.el8.x86_64/src/butil/memory/singleton_on_pthread_once.h:41
#8 0x00007f2e7024ae67 in __pthread_once_slow (once_control=0x7f2e7e67b8c0 <butil::GetLeakySingleton<bvar::ReadVersion>::g_create_leaky_singleton_once>, init_routine=0x7f2e7df5df90 <butil::GetLeakySingleton<bvar::ReadVersion>::create_leaky_singleton()>) at pthread_once.c:116
Thread 66 (Thread 0x7f2db97fa700 (LWP 3633582)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef01e10 in _int_free (av=0x7f2d90000020, p=0x7f2d900d39e0, have_lock=<optimized out>) at malloc.c:4314
Thread 63 (Thread 0x7f2dbaffd700 (LWP 3633579)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef04650 in __GI___libc_malloc (bytes=bytes@entry=7934) at malloc.c:3071
#2 0x00007f2e6ef89381 in __backtrace_symbols (array=0x7f2e405e7ad8, size=<optimized out>) at backtracesyms.c:69
#3 0x0000559de317aeba in BackTrace::BackTrace (this=0x7f2e405e7ad0, s=1) at BackTrace.h:29
#4 0x0000559de3178e7a in handle_fatal_signal (signum=6) at signal_handler.cc:167
#5 <signal handler called>
#6 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#7 0x00007f2e6eea0b25 in __GI_abort () at abort.c:79
#8 0x00007f2e6eef9897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f2e6f006057 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#9 0x00007f2e6eefffdc in malloc_printerr (str=str@entry=0x7f2e6f0080f8 "malloc(): invalid size (unsorted)") at malloc.c:5366
#10 0x00007f2e6ef02e6c in _int_malloc (av=av@entry=0x7f2d90000020, bytes=bytes@entry=40) at malloc.c:3748
#11 0x00007f2e6ef04662 in __GI___libc_malloc (bytes=40) at malloc.c:3073
#12 0x00007f2e6f871d3c in operator new(unsigned long) () from /lib64/libstdc++.so.6
#13 0x0000559de342482a in __gnu_cxx::new_allocator<std::_Rb_tree_node<unsigned long> >::allocate (this=0x7f2d901bf9a0, __n=1) at /usr/include/c++/8/ext/new_allocator.h:111
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] weixinwei closed issue #1311: glibc malloc deadlock
Posted by GitBox <gi...@apache.org>.
weixinwei closed issue #1311:
URL: https://github.com/apache/incubator-brpc/issues/1311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] cdjingit commented on issue #1311: glibc malloc deadlock
Posted by GitBox <gi...@apache.org>.
cdjingit commented on issue #1311:
URL: https://github.com/apache/incubator-brpc/issues/1311#issuecomment-750833645
malloc内存分配失败导致的死锁。malloc失败触发->backtrace->malloc导致死锁。参考:
https://sourceware.org/bugzilla/show_bug.cgi?id=16159
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] weixinwei edited a comment on issue #1311: glibc malloc deadlock
Posted by GitBox <gi...@apache.org>.
weixinwei edited a comment on issue #1311:
URL: https://github.com/apache/incubator-brpc/issues/1311#issuecomment-750838212
> malloc内存分配失败导致的死锁。malloc失败触发->backtrace->malloc导致死锁。参考:
> https://sourceware.org/bugzilla/show_bug.cgi?id=16159
您好，看您发的这个bug是2.12版本的。难道2.28版本还有这个问题
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] weixinwei closed issue #1311: glibc malloc deadlock
Posted by GitBox <gi...@apache.org>.
weixinwei closed issue #1311:
URL: https://github.com/apache/incubator-brpc/issues/1311
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] weixinwei commented on issue #1311: glibc malloc deadlock
Posted by GitBox <gi...@apache.org>.
weixinwei commented on issue #1311:
URL: https://github.com/apache/incubator-brpc/issues/1311#issuecomment-750838212
> malloc内存分配失败导致的死锁。malloc失败触发->backtrace->malloc导致死锁。参考:
> https://sourceware.org/bugzilla/show_bug.cgi?id=16159
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org