You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Jie Huang (Jira)" <ji...@apache.org> on 2022/11/17 01:58:00 UTC
[jira] [Commented] (KUDU-3405) Tabelt Server become 'Zombie Process'
[ https://issues.apache.org/jira/browse/KUDU-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17635074#comment-17635074 ]
Jie Huang commented on KUDU-3405:
---------------------------------
[~aserbin]
Thread 27 (Thread 0xffff635cef90 (LWP 10632)):
#0 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff635cab74, p=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
#1 sys_waitpid (options=1073741824, status=0xffff635cab74, pid=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3697
#2 google_breakpad::ExceptionHandler::GenerateDump (this=0x12d0c4d0, context=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:553
#3 0x00000000020838fc in google_breakpad::ExceptionHandler::SignalHandler (sig=11, info=0xffff635cac90, uc=0xffff635cad10)
at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:376
#4 <signal handler called>
#5 dwarf_get (c=0xffff635cce20, val=<synthetic pointer>, loc=...) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h:146
#6 apply_reg_state (c=c@entry=0xffff635cce20, rs=rs@entry=0xffff635cc718) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:878
#7 0x0000000002065dd8 in _ULaarch64_dwarf_step (c=c@entry=0xffff635cce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:967
#8 0x0000000002063ef4 in _ULaarch64_step (cursor=0xffff635cce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:146
#9 0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x75d33f18, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
#10 0x0000000001f8948c in Collect (skip_frames=1, this=0x75d33f18) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
#11 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
#12 <signal handler called>
#13 0x0000ffffb0b4f640 in clone () from /lib64/libc.so.6
#14 0x0000ffffb12f6d78 in do_clone.constprop.3 () from /lib64/libpthread.so.0
#15 0x0000ffffb12f84f8 in pthread_create@@GLIBC_2.17 () from /lib64/libpthread.so.0
#16 0x00000000020268bc in kudu::Thread::StartThread(std::string, std::string, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) (category=..., name=..., functor=..., flags=<error reading variable: Cannot access memory at address 0x69>,
holder=<error reading variable: Cannot access memory at address 0x89>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/thread.cc:618
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
===================
I don't think it was caused by the breakpad. Like above, "error(#12) -> breakpad-error(#4)"
> Tabelt Server become 'Zombie Process'
> -------------------------------------
>
> Key: KUDU-3405
> URL: https://issues.apache.org/jira/browse/KUDU-3405
> Project: Kudu
> Issue Type: Bug
> Components: tserver
> Affects Versions: 1.13.0
> Environment: linux-4.18.0-147.8.1.el7.aarch64
> gcc-4.8.5
> docker-19.03.4
> kudu-1.13.0(master*3，tserver*11)
> Reporter: Jie Huang
> Priority: Major
>
> In ARM, running KUDU. Tablet server incidental 'Zombie Process':
> 1. Process is exist;
> 2.Port 7050/8050 exist;
> 3.Log print is stopped;
> 4.minidmp is printed;
> 5.other tserver failed to connect it;
> 6.master failed to connect it, and loss of heartbeat;
> 7.memory/cpu/disk is normal;
>
> minidmp:
> {quote}${KUDU_HOME}/thirdparty/installed/uninstrumented/bin/minidump-2-core -o xxx.core xxx.dmp
> gdb bin/kudu-tserver xxx.core
> (gdb) bt
> {quote}
>
> {quote}GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "aarch64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /opt/kudu-1.13.0-aarch64/kudu-1.13.0-aarch64/release/bin/kudu-tserver...done.
> warning: core file may not match specified executable file.
> [New LWP 44946]
> [New LWP 7]
> [New LWP 9]
> [New LWP 10]
> [New LWP 11]
> [New LWP 12]
> [New LWP 13]
> [New LWP 14]
> [New LWP 85]
> [New LWP 86]
> [New LWP 87]
> [New LWP 88]
> [New LWP 89]
> [New LWP 90]
> [New LWP 91]
> [New LWP 92]
> [New LWP 93]
> [New LWP 148]
> [New LWP 149]
> [New LWP 150]
> [New LWP 151]
> [New LWP 152]
> [New LWP 153]
> [New LWP 154]
> [New LWP 155]
> [New LWP 156]
> [New LWP 157]
> [New LWP 158]
> [New LWP 159]
> [New LWP 160]
> [New LWP 161]
> [New LWP 162]
> [New LWP 163]
> [New LWP 164]
> [New LWP 165]
> [New LWP 166]
> [New LWP 167]
> [New LWP 168]
> [New LWP 169]
> [New LWP 170]
> [New LWP 171]
> [New LWP 172]
> [New LWP 173]
> [New LWP 174]
> [New LWP 175]
> [New LWP 176]
> [New LWP 177]
> [New LWP 178]
> [New LWP 179]
> [New LWP 180]
> [New LWP 181]
> [New LWP 182]
> [New LWP 183]
> [New LWP 184]
> [New LWP 185]
> [New LWP 186]
> [New LWP 187]
> [New LWP 188]
> [New LWP 189]
> [New LWP 190]
> [New LWP 191]
> [New LWP 192]
> [New LWP 193]
> [New LWP 194]
> [New LWP 195]
> [New LWP 196]
> [New LWP 197]
> [New LWP 198]
> [New LWP 199]
> [New LWP 200]
> [New LWP 201]
> [New LWP 202]
> [New LWP 203]
> [New LWP 204]
> [New LWP 205]
> [New LWP 206]
> [New LWP 207]
> [New LWP 208]
> [New LWP 209]
> [New LWP 210]
> [New LWP 211]
> [New LWP 212]
> [New LWP 213]
> [New LWP 214]
> [New LWP 215]
> [New LWP 216]
> [New LWP 217]
> [New LWP 218]
> [New LWP 219]
> [New LWP 220]
> [New LWP 221]
> [New LWP 222]
> [New LWP 223]
> [New LWP 224]
> [New LWP 225]
> [New LWP 226]
> [New LWP 227]
> [New LWP 228]
> [New LWP 229]
> [New LWP 230]
> [New LWP 231]
> [New LWP 232]
> [New LWP 233]
> [New LWP 234]
> [New LWP 235]
> [New LWP 236]
> [New LWP 237]
> [New LWP 238]
> [New LWP 239]
> [New LWP 240]
> [New LWP 241]
> [New LWP 242]
> [New LWP 243]
> [New LWP 244]
> [New LWP 245]
> [New LWP 246]
> [New LWP 247]
> [New LWP 248]
> [New LWP 249]
> [New LWP 250]
> [New LWP 251]
> [New LWP 252]
> [New LWP 253]
> [New LWP 254]
> [New LWP 255]
> [New LWP 884]
> [New LWP 44893]
> [New LWP 44901]
> [New LWP 44919]
> [New LWP 44922]
> [New LWP 44923]
> [New LWP 44924]
> [New LWP 44925]
> [New LWP 44926]
> [New LWP 44927]
> [New LWP 44928]
> [New LWP 44929]
> [New LWP 44930]
> [New LWP 44931]
> [New LWP 44932]
> [New LWP 44933]
> [New LWP 44934]
> [New LWP 44935]
> [New LWP 44936]
> [New LWP 44937]
> [New LWP 44938]
> [New LWP 44939]
> [New LWP 44940]
> [New LWP 44941]
> [New LWP 44942]
> [New LWP 44943]
> [New LWP 44944]
> [New LWP 44945]
> [New LWP 44947]
> warning: Could not load shared library symbols for linux-gate.so.
> Do you need "set solib-search-path" or "set sysroot"?
> Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error
> warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
> To enable execution of this file add
> add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so
> line to your configuration file "/root/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/root/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error
> warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Failed to read a valid object file image from memory.
> Core was generated by `/kudu/release/bin/kudu-tserver --flagfile=/opt/kudu/conf/tserver.'.
> Program terminated with signal 11, Segmentation fault.
> #0 dwarf_get (c=0xffffa43dce20, val=<synthetic pointer>, loc=...) at ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h:146
> 146 ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h: No such file or directory.
> Missing separate debuginfos, use: debuginfo-install cyrus-sasl-gs2-2.1.26-23.el7.aarch64 cyrus-sasl-gssapi-2.1.26-23.el7.aarch64 cyrus-sasl-ldap-2.1.26-23.el7.aarch64 cyrus-sasl-lib-2.1.26-23.el7.aarch64 cyrus-sasl-md5-2.1.26-23.el7.aarch64 cyrus-sasl-ntlm-2.1.26-23.el7.aarch64 cyrus-sasl-plain-2.1.26-23.el7.aarch64 cyrus-sasl-scram-2.1.26-23.el7.aarch64 cyrus-sasl-sql-2.1.26-23.el7.aarch64 glibc-2.17-317.el7.aarch64 keyutils-libs-1.5.8-3.el7.aarch64 krb5-libs-1.15.1-50.el7.aarch64 libcom_err-1.42.9-19.el7.aarch64 libdb-5.3.21-25.el7.aarch64 libgcc-4.8.5-44.el7.aarch64 libselinux-2.5-15.el7.aarch64 libstdc++-4.8.5-39.el7.aarch64 mariadb-libs-5.5.68-1.el7.aarch64 ncurses-libs-5.9-14.20130511.el7_4.aarch64 nspr-4.21.0-1.el7.aarch64 nss-3.44.0-7.el7_7.aarch64 nss-softokn-freebl-3.44.0-8.el7_7.aarch64 nss-util-3.44.0-4.el7_7.aarch64 openldap-2.4.44-21.el7_6.aarch64 openssl-libs-1.0.2k-19.el7.aarch64 pcre-8.32-17.el7.aarch64 postgresql-libs-9.2.24-4.el7_8.aarch64 zlib-1.2.7-18.el7.aarch64
> (gdb) bt
> #0 dwarf_get (c=0xffffa43dce20, val=<synthetic pointer>, loc=...) at ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h:146
> #1 apply_reg_state (c=0xffffa43dce20, c@entry=<error reading variable: Cannot access memory at address 0xffffa43dbef8>, rs=<error reading variable: Cannot access memory at address 0xffffa43dbf68>)
> at ${KUDU_HOME}/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:878
> (gdb)
> {quote}
> gdb -p [pid]: (Extract section)
> {quote}gdb -p [pid]
> (gdb) info threads
> (gdb) thread apply all bt
> {quote}
> case 1:
> {quote}...
> 148 Thread 0xffffa79def90 (LWP 314) "excess-log-dele" 0x0000ffffb2ecf008 in __lll_lock_wait () from /lib64/libpthread.so.0
> ...
> 13 Thread 0xffff9e0aef90 (LWP 14437) "raft [worker]-1" 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff9e0ab6f4, p=<optimized out>)
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
> ...
>
> ...
> Thread 13 (Thread 0xffff9e0aef90 (LWP 14437)):
> #0 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff9e0ab6f4, p=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
> #1 sys_waitpid (options=1073741824, status=0xffff9e0ab6f4, pid=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3697
> #2 google_breakpad::ExceptionHandler::GenerateDump (this=0x3515c4d0, context=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:553
> #3 0x00000000020838fc in google_breakpad::ExceptionHandler::SignalHandler (sig=11, info=0xffff9e0ab810, uc=0xffff9e0ab890)
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:376
> #4 <signal handler called>
> #5 access_mem (as=0x2dc6220 <local_addr_space>, addr=1, val=0xffff9e0aca98, write=0, arg=0xffff9e0acb40) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Ginit.c:96
> #6 0x0000000002066b6c in _ULaarch64_is_signal_frame (cursor=cursor@entry=0xffff9e0ace20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gis_signal_frame.c:51
> #7 0x0000000002063d18 in _ULaarch64_step (cursor=0xffff9e0ace20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:143
> #8 0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x54007dd8, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
> #9 0x0000000001f8948c in Collect (skip_frames=1, this=0x54007dd8) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
> #10 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
> #11 <signal handler called>
> #12 0x0000ffffb271f640 in clone () from /lib64/libc.so.6
> #13 0x0000ffffb2ec6d78 in do_clone.constprop.3 () from /lib64/libpthread.so.0
> #14 0x0000ffffb2ec84f8 in pthread_create@@GLIBC_2.17 () from /lib64/libpthread.so.0
> #15 0x0000000000000001 in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> ...
> Thread 148 (Thread 0xffffa79def90 (LWP 314)):
> #0 0x0000ffffb2ecf008 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x0000ffffb2ec9c40 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #2 0x0000000002065c14 in get_rs_cache (saved_maskp=0xffffa79dbf60, as=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:610
> #3 find_reg_state (sr=0xffffa79dbfe0, c=0xffffa79dca80) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:908
> #4 _ULaarch64_dwarf_step (c=c@entry=0xffffa79dca80) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:965
> #5 0x0000000002063ef4 in _ULaarch64_step (cursor=0xffffa79dca80) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:146
> #6 0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x54002838, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
> #7 0x0000000001f8948c in Collect (skip_frames=1, this=0x54002838) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
> #8 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
> #9 <signal handler called>
> #10 0x0000ffffb2ecc420 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> #11 0x0000000001f88bc8 in kudu::ConditionVariable::WaitUntil (this=this@entry=0xffffcf707510, until=...) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/condition_variable.cc:87
> #12 0x0000000000a80f54 in WaitUntil (when=..., this=0xffffcf7074e0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/countdown_latch.h:87
> #13 kudu::server::ServerBase::ExcessLogFileDeleterThread (this=0xffffcf707268) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/server/server_base.cc:721
> #14 0x0000000002029b3c in operator() (this=0x39c67228) at /usr/include/c++/4.8.2/functional:2471
> #15 kudu::Thread::SuperviseThread (arg=0x39c67200) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/thread.cc:674
> #16 0x0000ffffb2ec7d38 in start_thread () from /lib64/libpthread.so.0
> #17 0x0000ffffb271f690 in thread_start () from /lib64/libc.so.6
> ...
> {quote}
>
> case 2:
> {quote} 155 Thread 0xffff93fcef90 (LWP 9) "kudu-tserver" SLL_SetNext (n=0x0, t=0x36e28fc0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/gperftools-2.6.90/src/linked_list.h:49
> 154 Thread 0xffff91ecef90 (LWP 10) "kudu-tserver" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 153 Thread 0xffff916bef90 (LWP 11) "kudu-tserver" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 152 Thread 0xffff90eaef90 (LWP 12) "kernel-watcher-" 0x0000ffff9456cba0 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /lib64/libstdc++.so.6
> 151 Thread 0xffff9069ef90 (LWP 13) "sigusr1-handler" 0x0000ffff94ab0708 in sigwait () from /lib64/libpthread.so.0
> 150 Thread 0xffff8fe8ef90 (LWP 14) "file cache-evic" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 149 Thread 0xffff8ce2ef90 (LWP 511) "rpc reactor-511" _Function_base (this=0x1191ee450) at /usr/include/c++/4.8.2/functional:2025
> 148 Thread 0xffff8b5fef90 (LWP 512) "rpc reactor-512" 0x000000000206b934 in downheap (heap=0xbaae5000, N=240, k=4) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libev-4.20/ev.c:2141
> 147 Thread 0xffff8be0ef90 (LWP 513) "rpc reactor-513" <lambda> (this=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/consensus/consensus_peers.cc:188
> 146 Thread 0xffff8e65ef90 (LWP 514) "rpc reactor-514" 0x0000000000bb0300 in _M_add_ref_lock (this=0x7869f900) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:246
> 145 Thread 0xffff8f25ef90 (LWP 515) "diag-logger-515" 0x0000000001f8a8b4 in sys_futex (u2=0x0, v2=0, t=0xffff8f25e120, v=0, o=128, u=0x43647cf0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/gutil/linux_syscall_support.h:3401
> 144 Thread 0xffff8de4ef90 (LWP 516) "result-tracker-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 143 Thread 0xffff8d63ef90 (LWP 517) "excess-log-dele" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 142 Thread 0xffff8c61ef90 (LWP 518) "tcmalloc-memory" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 141 Thread 0xffff8adeef90 (LWP 519) "MaintenanceMgr " 0x0000ffff9425543c in sigprocmask () from /lib64/libc.so.6
> 140 Thread 0xffff8a5def90 (LWP 520) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
> 139 Thread 0xffff89dcef90 (LWP 521) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
> 138 Thread 0xffff895bef90 (LWP 522) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
> 137 Thread 0xffff88daef90 (LWP 523) "MaintenanceMgr " 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
> 136 Thread 0xffff7dc4ef90 (LWP 545) "removal_thread-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 135 Thread 0xffff7d43ef90 (LWP 546) "tc-session-exp-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 134 Thread 0xffff7cc2ef90 (LWP 547) "rpc worker-547" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 133 Thread 0xffff7c41ef90 (LWP 548) "rpc worker-548" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 132 Thread 0xffff7bc0ef90 (LWP 549) "rpc worker-549" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 131 Thread 0xffff7b3fef90 (LWP 550) "rpc worker-550" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 130 Thread 0xffff7abeef90 (LWP 551) "rpc worker-551" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 129 Thread 0xffff7a3def90 (LWP 552) "rpc worker-552" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 128 Thread 0xffff79bcef90 (LWP 553) "rpc worker-553" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 127 Thread 0xffff793bef90 (LWP 554) "rpc worker-554" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 126 Thread 0xffff78baef90 (LWP 555) "rpc worker-555" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 125 Thread 0xffff7839ef90 (LWP 556) "rpc worker-556" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 124 Thread 0xffff77b8ef90 (LWP 557) "rpc worker-557" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 123 Thread 0xffff7737ef90 (LWP 558) "rpc worker-558" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 122 Thread 0xffff76b6ef90 (LWP 559) "rpc worker-559" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 121 Thread 0xffff7635ef90 (LWP 560) "rpc worker-560" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 120 Thread 0xffff75b4ef90 (LWP 561) "rpc worker-561" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 119 Thread 0xffff7533ef90 (LWP 562) "rpc worker-562" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 118 Thread 0xffff74b2ef90 (LWP 563) "rpc worker-563" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 117 Thread 0xffff7431ef90 (LWP 564) "rpc worker-564" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 116 Thread 0xffff73b0ef90 (LWP 565) "rpc worker-565" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 115 Thread 0xffff732fef90 (LWP 566) "rpc worker-566" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 114 Thread 0xffff72aeef90 (LWP 567) "rpc worker-567" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 113 Thread 0xffff722def90 (LWP 568) "rpc worker-568" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 112 Thread 0xffff71acef90 (LWP 569) "rpc worker-569" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 111 Thread 0xffff70aaef90 (LWP 570) "rpc worker-570" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 110 Thread 0xffff7029ef90 (LWP 572) "rpc worker-572" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 109 Thread 0xffff6fa8ef90 (LWP 573) "rpc worker-573" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 108 Thread 0xffff6f27ef90 (LWP 574) "rpc worker-574" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 107 Thread 0xffff6ea6ef90 (LWP 575) "rpc worker-575" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 106 Thread 0xffff6e25ef90 (LWP 576) "rpc worker-576" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 105 Thread 0xffff6da4ef90 (LWP 577) "rpc worker-577" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 104 Thread 0xffff6d23ef90 (LWP 578) "rpc worker-578" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 103 Thread 0xffff6ca2ef90 (LWP 579) "rpc worker-579" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 102 Thread 0xffff6c21ef90 (LWP 580) "rpc worker-580" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 101 Thread 0xffff6ba0ef90 (LWP 581) "rpc worker-581" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 100 Thread 0xffff6b1fef90 (LWP 582) "rpc worker-582" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 99 Thread 0xffff6a9eef90 (LWP 583) "rpc worker-583" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 98 Thread 0xffff6a1def90 (LWP 584) "rpc worker-584" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 97 Thread 0xffff699cef90 (LWP 585) "rpc worker-585" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 96 Thread 0xffff691bef90 (LWP 586) "rpc worker-586" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 95 Thread 0xffff689aef90 (LWP 587) "rpc worker-587" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 94 Thread 0xffff6819ef90 (LWP 588) "rpc worker-588" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 93 Thread 0xffff6798ef90 (LWP 589) "rpc worker-589" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 92 Thread 0xffff6717ef90 (LWP 590) "rpc worker-590" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 91 Thread 0xffff6696ef90 (LWP 591) "rpc worker-591" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 90 Thread 0xffff6615ef90 (LWP 592) "rpc worker-592" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 89 Thread 0xffff6594ef90 (LWP 593) "rpc worker-593" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 88 Thread 0xffff6513ef90 (LWP 594) "rpc worker-594" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 87 Thread 0xffff6492ef90 (LWP 595) "rpc worker-595" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 86 Thread 0xffff6411ef90 (LWP 596) "rpc worker-596" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 85 Thread 0xffff6390ef90 (LWP 597) "rpc worker-597" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 84 Thread 0xffff630fef90 (LWP 598) "rpc worker-598" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 83 Thread 0xffff628eef90 (LWP 599) "rpc worker-599" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 82 Thread 0xffff620def90 (LWP 600) "rpc worker-600" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 81 Thread 0xffff618cef90 (LWP 601) "rpc worker-601" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 80 Thread 0xffff610bef90 (LWP 602) "rpc worker-602" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 79 Thread 0xffff608aef90 (LWP 603) "rpc worker-603" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 78 Thread 0xffff6009ef90 (LWP 604) "rpc worker-604" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 77 Thread 0xffff5f88ef90 (LWP 605) "rpc worker-605" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 76 Thread 0xffff5f07ef90 (LWP 606) "rpc worker-606" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 75 Thread 0xffff5e86ef90 (LWP 607) "rpc worker-607" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 74 Thread 0xffff5e05ef90 (LWP 608) "rpc worker-608" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 73 Thread 0xffff5d84ef90 (LWP 609) "rpc worker-609" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 72 Thread 0xffff5d03ef90 (LWP 610) "rpc worker-610" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 71 Thread 0xffff5c82ef90 (LWP 611) "rpc worker-611" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 70 Thread 0xffff5c01ef90 (LWP 612) "rpc worker-612" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 69 Thread 0xffff5b80ef90 (LWP 613) "rpc worker-613" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 68 Thread 0xffff5affef90 (LWP 614) "rpc worker-614" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 67 Thread 0xffff5a7eef90 (LWP 615) "rpc worker-615" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 66 Thread 0xffff59fdef90 (LWP 616) "rpc worker-616" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 65 Thread 0xffff597cef90 (LWP 617) "rpc worker-617" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 64 Thread 0xffff58fbef90 (LWP 618) "rpc worker-618" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 63 Thread 0xffff587aef90 (LWP 619) "rpc worker-619" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 62 Thread 0xffff57f9ef90 (LWP 620) "rpc worker-620" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 61 Thread 0xffff5778ef90 (LWP 621) "rpc worker-621" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 60 Thread 0xffff56f7ef90 (LWP 622) "rpc worker-622" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 59 Thread 0xffff5676ef90 (LWP 623) "rpc worker-623" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 58 Thread 0xffff55f5ef90 (LWP 624) "rpc worker-624" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 57 Thread 0xffff5574ef90 (LWP 625) "rpc worker-625" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 56 Thread 0xffff54f3ef90 (LWP 626) "rpc worker-626" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 55 Thread 0xffff5472ef90 (LWP 627) "rpc worker-627" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 54 Thread 0xffff53f1ef90 (LWP 628) "rpc worker-628" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 53 Thread 0xffff5370ef90 (LWP 629) "rpc worker-629" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 52 Thread 0xffff52efef90 (LWP 630) "rpc worker-630" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 51 Thread 0xffff526eef90 (LWP 631) "rpc worker-631" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 50 Thread 0xffff51edef90 (LWP 632) "rpc worker-632" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 49 Thread 0xffff516cef90 (LWP 633) "rpc worker-633" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 48 Thread 0xffff50ebef90 (LWP 634) "rpc worker-634" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 47 Thread 0xffff506aef90 (LWP 635) "rpc worker-635" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 46 Thread 0xffff4fe9ef90 (LWP 636) "rpc worker-636" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 45 Thread 0xffff4f68ef90 (LWP 637) "rpc worker-637" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 44 Thread 0xffff4ee7ef90 (LWP 638) "rpc worker-638" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 43 Thread 0xffff4e66ef90 (LWP 639) "rpc worker-639" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 42 Thread 0xffff4de5ef90 (LWP 640) "rpc worker-640" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 41 Thread 0xffff4d64ef90 (LWP 641) "rpc worker-641" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 40 Thread 0xffff4ce3ef90 (LWP 642) "rpc worker-642" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 39 Thread 0xffff4c62ef90 (LWP 643) "rpc worker-643" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 38 Thread 0xffff4be1ef90 (LWP 644) "rpc worker-644" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 37 Thread 0xffff4b60ef90 (LWP 645) "rpc worker-645" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 36 Thread 0xffff4adfef90 (LWP 646) "rpc worker-646" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 35 Thread 0xffff4a5eef90 (LWP 647) "rpc worker-647" 0x0000ffff94aac0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 34 Thread 0xffff49ddef90 (LWP 648) "acceptor-648" 0x0000ffff94300780 in accept4 () from /lib64/libc.so.6
> 33 Thread 0xffff495cef90 (LWP 649) "sq_acceptor" 0x0000ffff942f4e24 in poll () from /lib64/libc.so.6
> 32 Thread 0xffff48dbef90 (LWP 650) "heartbeat-650" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 31 Thread 0xffff485aef90 (LWP 651) "heartbeat-651" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 30 Thread 0xffff47d9ef90 (LWP 652) "heartbeat-652" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 29 Thread 0xffff4758ef90 (LWP 653) "maintenance_sch" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 28 Thread 0xffff208bef90 (LWP 1449) "sq_worker" 0x0000ffff942f4e24 in poll () from /lib64/libc.so.6
> 27 Thread 0xffff813aef90 (LWP 24789) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 26 Thread 0xffff7fb7ef90 (LWP 24886) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 25 Thread 0xffff8038ef90 (LWP 24911) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 24 Thread 0xffff44d3ef90 (LWP 24912) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 23 Thread 0xffff424eef90 (LWP 24913) "prepare [worker" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 22 Thread 0xffff42cfef90 (LWP 24914) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 21 Thread 0xffff4350ef90 (LWP 24915) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 20 Thread 0xffff874fef90 (LWP 24916) "prepare [worker" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 19 Thread 0xffff86ceef90 (LWP 24917) "apply [worker]-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 18 Thread 0xffff864def90 (LWP 24918) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 17 Thread 0xffff85ccef90 (LWP 24919) "apply [worker]-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 16 Thread 0xffff854bef90 (LWP 24920) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 15 Thread 0xffff84c1ef90 (LWP 24921) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 14 Thread 0xffff8440ef90 (LWP 24922) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 13 Thread 0xffff83bfef90 (LWP 24923) "raft [worker]-2" 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff83bfb6f4, p=<optimized out>)
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
> 12 Thread 0xffff833eef90 (LWP 24924) "raft [worker]-2" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 11 Thread 0xffff82bdef90 (LWP 24925) "apply [worker]-" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 10 Thread 0xffff823cef90 (LWP 24926) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 9 Thread 0xffff81bbef90 (LWP 24927) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 8 Thread 0xffff80b9ef90 (LWP 24928) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 7 Thread 0xffff7f36ef90 (LWP 24929) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 6 Thread 0xffff7eb5ef90 (LWP 24930) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 5 Thread 0xffff712bef90 (LWP 24931) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 4 Thread 0xffff46d7ef90 (LWP 24932) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 3 Thread 0xffff4656ef90 (LWP 24933) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 2 Thread 0xffff45d5ef90 (LWP 24935) "wal-append [wor" 0x0000ffff94aac41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> * 1 Thread 0xffff94bdb450 (LWP 7) "kudu-tserver" 0x0000ffff94aafcb4 in nanosleep () from /lib64/libpthread.so.0
> ...
> Thread 13 (Thread 0xffff83bfef90 (LWP 24923)):
> #0 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff83bfb6f4, p=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
> #1 sys_waitpid (options=1073741824, status=0xffff83bfb6f4, pid=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3697
> #2 google_breakpad::ExceptionHandler::GenerateDump (this=0x1a03e4d0, context=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:553
> #3 0x00000000020838fc in google_breakpad::ExceptionHandler::SignalHandler (sig=11, info=0xffff83bfb810, uc=0xffff83bfb890)
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:376
> #4 <signal handler called>
> #5 access_mem (as=0x2dc6220 <local_addr_space>, addr=1, val=0xffff83bfca98, write=0, arg=0xffff83bfcb40) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Ginit.c:96
> #6 0x0000000002066b6c in _ULaarch64_is_signal_frame (cursor=cursor@entry=0xffff83bfce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gis_signal_frame.c:51
> #7 0x0000000002063d18 in _ULaarch64_step (cursor=0xffff83bfce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:143
> #8 0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x38073978, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
> #9 0x0000000001f8948c in Collect (skip_frames=1, this=0x38073978) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
> #10 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
> #11 <signal handler called>
> #12 0x0000ffff942ff640 in clone () from /lib64/libc.so.6
> #13 0x0000ffff94aa6d78 in do_clone.constprop.3 () from /lib64/libpthread.so.0
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> Thread 141 (Thread 0xffff8adeef90 (LWP 519)):
> #0 0x0000ffff9425543c in sigprocmask () from /lib64/libc.so.6
> #1 0x0000000002065dc8 in put_rs_cache (as=<optimized out>, saved_maskp=0xffff8adead70, cache=0x2dc6280 <local_addr_space+96>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:634
> #2 find_reg_state (sr=0xffff8adeadf0, c=0xffff8adeb890) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:949
> #3 _ULaarch64_dwarf_step (c=c@entry=0xffff8adeb890) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:965
> #4 0x0000000002063ef4 in _ULaarch64_step (cursor=0xffff8adeb890) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:146
> #5 0x0000000001f88f6c in kudu::StackTrace::Collect (this=this@entry=0x3806e978, skip_frames=skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
> #6 0x0000000001f8948c in Collect (skip_frames=1, this=0x3806e978) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
> #7 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
> #8 <signal handler called>
> #9 0x0000ffff942f6aa4 in fdatasync () from /lib64/libc.so.6
> #10 0x0000000001fa9dec in kudu::(anonymous namespace)::DoSync (fd=4614, filename="/opt/java/kudu-1.13.0-aarch64/kudu-1.13.0-aarch64/tserver/data/7/data/80eb86ffb4af46b397440586d926cce8.data")
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/env_posix.cc:346
> #11 0x0000000001faa814 in kudu::(anonymous namespace)::PosixRWFile::Sync (this=0x1e03b580) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/env_posix.cc:983
> #12 0x0000000001fba228 in kudu::internal::Descriptor<kudu::RWFile>::Sync (this=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/file_cache.cc:293
> #13 0x0000000001daae1c in kudu::fs::internal::LogBlockContainer::SyncData (this=0x1d9a0dc0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1213
> #14 0x0000000001db46bc in kudu::fs::internal::LogBlockContainer::__lambda8::operator() (__closure=__closure@entry=0xffff8aded768) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1105
> #15 0x0000000001db516c in DoCloseBlocks (mode=kudu::fs::internal::LogBlockContainer::SYNC, blocks=std::vector of length 1, capacity 1 = \{...}, this=<optimized out>, this=<optimized out>)
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1129
> #16 kudu::fs::internal::LogBlockCreationTransaction::CommitCreatedBlocks (this=0x4bb325a0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/fs/log_block_manager.cc:1416
> #17 0x0000000000b402c0 in kudu::tablet::RollingDiskRowSetWriter::Finish (this=this@entry=0xffff8adedde8) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/diskrowset.cc:487
> #18 0x0000000000af8f04 in kudu::tablet::Tablet::DoMergeCompactionOrFlush (this=this@entry=0x7d614a80, input=..., mrs_being_flushed=mrs_being_flushed@entry=-1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/tablet.cc:1608
> #19 0x0000000000afb6d0 in kudu::tablet::Tablet::Compact (this=0x7d614a80, flags=flags@entry=0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/tablet.cc:1858
> #20 0x0000000000b1263c in kudu::tablet::CompactRowSetsOp::Perform (this=0x1b3067e0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/tablet/tablet_mm_ops.cc:176
> #21 0x0000000001fd9060 in kudu::MaintenanceManager::LaunchOp (this=0x2153f878, op=0x1b3067e0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/maintenance_manager.cc:545
> #22 0x0000000002031798 in operator() (this=0xffff8adee628) at /usr/include/c++/4.8.2/functional:2471
> #23 kudu::ThreadPool::DispatchThread (this=0x1f9345a0) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/threadpool.cc:662
> #24 0x0000000002029b3c in operator() (this=0x1ce37768) at /usr/include/c++/4.8.2/functional:2471
> #25 kudu::Thread::SuperviseThread (arg=0x1ce37740) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/thread.cc:674
> #26 0x0000ffff94aa7d38 in start_thread () from /lib64/libpthread.so.0
> #27 0x0000ffff942ff690 in thread_start () from /lib64/libc.so.6
> {quote}
>
> case 3:
> {quote} 178 Thread 0xffffb081ef90 (LWP 9) "kudu-tserver" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 177 Thread 0xffffae71ef90 (LWP 10) "kudu-tserver" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 176 Thread 0xffffadf0ef90 (LWP 11) "kudu-tserver" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 175 Thread 0xffffad6fef90 (LWP 12) "kernel-watcher-" 0x0000ffffb0ae8898 in __GI__IO_link_in () from /lib64/libc.so.6
> 174 Thread 0xffffaceeef90 (LWP 13) "sigusr1-handler" 0x0000ffffb1300708 in sigwait () from /lib64/libpthread.so.0
> 173 Thread 0xffffac6def90 (LWP 14) "file cache-evic" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 172 Thread 0xffff9b4bef90 (LWP 648) "rpc reactor-648" __weak_ptr (this=<optimized out>) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:1195
> 171 Thread 0xffff8caeef90 (LWP 649) "rpc reactor-649" _Hash_node_base (this=<optimized out>) at /usr/include/c++/4.8.2/bits/hashtable_policy.h:153
> 170 Thread 0xffff94beef90 (LWP 650) "rpc reactor-650" _Function_base (this=0x2eeb63c0) at /usr/include/c++/4.8.2/functional:2025
> 169 Thread 0xffffa8e6ef90 (LWP 651) "rpc reactor-651" 0x0000ffffb0b65030 in clock_gettime () from /lib64/libc.so.6
> 168 Thread 0xffffabaaef90 (LWP 652) "diag-logger-652" 0x0000000001f8a8b4 in sys_futex (u2=0x0, v2=0, t=0xffffabaae120, v=0, o=128, u=0x191ad350) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/gutil/linux_syscall_support.h:3401
> 167 Thread 0xffffab29ef90 (LWP 653) "result-tracker-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 166 Thread 0xffffaaa8ef90 (LWP 654) "excess-log-dele" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 165 Thread 0xffffaa27ef90 (LWP 655) "tcmalloc-memory" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 164 Thread 0xffffa9a6ef90 (LWP 656) "MaintenanceMgr " 0x0000ffffb0b46aa4 in fdatasync () from /lib64/libc.so.6
> 163 Thread 0xffff9dd0ef90 (LWP 679) "removal_thread-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 162 Thread 0xffff9d4fef90 (LWP 680) "tc-session-exp-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 161 Thread 0xffff9cceef90 (LWP 681) "rpc worker-681" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 160 Thread 0xffff9c4def90 (LWP 682) "rpc worker-682" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 159 Thread 0xffff9bccef90 (LWP 683) "rpc worker-683" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 158 Thread 0xffff9acaef90 (LWP 684) "rpc worker-684" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 157 Thread 0xffff9a49ef90 (LWP 685) "rpc worker-685" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 156 Thread 0xffff99c8ef90 (LWP 686) "rpc worker-686" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 155 Thread 0xffff9947ef90 (LWP 687) "rpc worker-687" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 154 Thread 0xffff98c6ef90 (LWP 688) "rpc worker-688" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 153 Thread 0xffff9845ef90 (LWP 689) "rpc worker-689" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 152 Thread 0xffff97c4ef90 (LWP 690) "rpc worker-690" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 151 Thread 0xffff96c2ef90 (LWP 691) "rpc worker-691" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 150 Thread 0xffff9641ef90 (LWP 692) "rpc worker-692" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 149 Thread 0xffff95c0ef90 (LWP 693) "rpc worker-693" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 148 Thread 0xffff953fef90 (LWP 694) "rpc worker-694" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 147 Thread 0xffff943def90 (LWP 695) "rpc worker-695" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 146 Thread 0xffff93bcef90 (LWP 696) "rpc worker-696" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 145 Thread 0xffff933bef90 (LWP 697) "rpc worker-697" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 144 Thread 0xffff92baef90 (LWP 698) "rpc worker-698" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 143 Thread 0xffff9239ef90 (LWP 699) "rpc worker-699" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 142 Thread 0xffff91b8ef90 (LWP 700) "rpc worker-700" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 141 Thread 0xffff9137ef90 (LWP 701) "rpc worker-701" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 140 Thread 0xffff90b6ef90 (LWP 702) "rpc worker-702" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 139 Thread 0xffff9035ef90 (LWP 703) "rpc worker-703" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 138 Thread 0xffff8fb4ef90 (LWP 704) "rpc worker-704" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 137 Thread 0xffff8f33ef90 (LWP 705) "rpc worker-705" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 136 Thread 0xffff8eb2ef90 (LWP 706) "rpc worker-706" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 135 Thread 0xffff8e31ef90 (LWP 707) "rpc worker-707" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 134 Thread 0xffff8db0ef90 (LWP 708) "rpc worker-708" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 133 Thread 0xffff8d2fef90 (LWP 709) "rpc worker-709" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 132 Thread 0xffff8c2def90 (LWP 710) "rpc worker-710" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 131 Thread 0xffff8bacef90 (LWP 711) "rpc worker-711" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 130 Thread 0xffff8b2bef90 (LWP 712) "rpc worker-712" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 129 Thread 0xffff8aaaef90 (LWP 713) "rpc worker-713" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 128 Thread 0xffff8a29ef90 (LWP 714) "rpc worker-714" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 127 Thread 0xffff89a8ef90 (LWP 715) "rpc worker-715" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 126 Thread 0xffff8927ef90 (LWP 716) "rpc worker-716" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 125 Thread 0xffff88a6ef90 (LWP 717) "rpc worker-717" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 124 Thread 0xffff8825ef90 (LWP 718) "rpc worker-718" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 123 Thread 0xffff87a4ef90 (LWP 719) "rpc worker-719" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 122 Thread 0xffff8723ef90 (LWP 720) "rpc worker-720" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 121 Thread 0xffff86a2ef90 (LWP 721) "rpc worker-721" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 120 Thread 0xffff8621ef90 (LWP 722) "rpc worker-722" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 119 Thread 0xffff85a0ef90 (LWP 723) "rpc worker-723" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 118 Thread 0xffff851fef90 (LWP 724) "rpc worker-724" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 117 Thread 0xffff849eef90 (LWP 725) "rpc worker-725" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 116 Thread 0xffff841def90 (LWP 726) "rpc worker-726" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 115 Thread 0xffff839cef90 (LWP 727) "rpc worker-727" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 114 Thread 0xffff831bef90 (LWP 728) "rpc worker-728" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 113 Thread 0xffff829aef90 (LWP 729) "rpc worker-729" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 112 Thread 0xffff8219ef90 (LWP 730) "rpc worker-730" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 111 Thread 0xffff8198ef90 (LWP 731) "rpc worker-731" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 110 Thread 0xffff8117ef90 (LWP 732) "rpc worker-732" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 109 Thread 0xffff8096ef90 (LWP 733) "rpc worker-733" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 108 Thread 0xffff8015ef90 (LWP 734) "rpc worker-734" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 107 Thread 0xffff7f94ef90 (LWP 735) "rpc worker-735" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 106 Thread 0xffff7f13ef90 (LWP 736) "rpc worker-736" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 105 Thread 0xffff7e92ef90 (LWP 737) "rpc worker-737" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 104 Thread 0xffff7e11ef90 (LWP 738) "rpc worker-738" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 103 Thread 0xffff7d90ef90 (LWP 739) "rpc worker-739" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 102 Thread 0xffff7d0fef90 (LWP 740) "rpc worker-740" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 101 Thread 0xffff7c8eef90 (LWP 741) "rpc worker-741" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 100 Thread 0xffff7c0def90 (LWP 742) "rpc worker-742" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 99 Thread 0xffff7b8cef90 (LWP 743) "rpc worker-743" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 98 Thread 0xffff7b0bef90 (LWP 744) "rpc worker-744" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 97 Thread 0xffff7a8aef90 (LWP 745) "rpc worker-745" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 96 Thread 0xffff7a09ef90 (LWP 746) "rpc worker-746" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 95 Thread 0xffff7988ef90 (LWP 747) "rpc worker-747" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 94 Thread 0xffff7907ef90 (LWP 748) "rpc worker-748" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 93 Thread 0xffff7886ef90 (LWP 749) "rpc worker-749" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 92 Thread 0xffff7805ef90 (LWP 750) "rpc worker-750" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 91 Thread 0xffff7784ef90 (LWP 751) "rpc worker-751" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 90 Thread 0xffff7703ef90 (LWP 752) "rpc worker-752" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 89 Thread 0xffff7682ef90 (LWP 753) "rpc worker-753" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 88 Thread 0xffff7601ef90 (LWP 754) "rpc worker-754" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 87 Thread 0xffff7580ef90 (LWP 755) "rpc worker-755" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 86 Thread 0xffff74ffef90 (LWP 756) "rpc worker-756" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 85 Thread 0xffff747eef90 (LWP 757) "rpc worker-757" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 84 Thread 0xffff73fdef90 (LWP 758) "rpc worker-758" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 83 Thread 0xffff737cef90 (LWP 759) "rpc worker-759" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 82 Thread 0xffff72fbef90 (LWP 760) "rpc worker-760" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 81 Thread 0xffff727aef90 (LWP 761) "rpc worker-761" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 80 Thread 0xffff71f9ef90 (LWP 762) "rpc worker-762" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 79 Thread 0xffff7178ef90 (LWP 763) "rpc worker-763" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 78 Thread 0xffff70f7ef90 (LWP 764) "rpc worker-764" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 77 Thread 0xffff7076ef90 (LWP 765) "rpc worker-765" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 76 Thread 0xffff6ff5ef90 (LWP 766) "rpc worker-766" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 75 Thread 0xffff6f74ef90 (LWP 767) "rpc worker-767" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 74 Thread 0xffff6ef3ef90 (LWP 768) "rpc worker-768" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 73 Thread 0xffff6e72ef90 (LWP 769) "rpc worker-769" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 72 Thread 0xffff6df1ef90 (LWP 770) "rpc worker-770" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 71 Thread 0xffff6d70ef90 (LWP 771) "rpc worker-771" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 70 Thread 0xffff6cefef90 (LWP 772) "rpc worker-772" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 69 Thread 0xffff6c6eef90 (LWP 773) "rpc worker-773" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 68 Thread 0xffff6bedef90 (LWP 774) "rpc worker-774" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 67 Thread 0xffff6b6cef90 (LWP 775) "rpc worker-775" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 66 Thread 0xffff6aebef90 (LWP 776) "rpc worker-776" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 65 Thread 0xffff6a6aef90 (LWP 777) "rpc worker-777" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 64 Thread 0xffff69e9ef90 (LWP 778) "rpc worker-778" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 63 Thread 0xffff6968ef90 (LWP 779) "rpc worker-779" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 62 Thread 0xffff68e7ef90 (LWP 780) "rpc worker-780" 0x0000ffffb12fc0f8 in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 61 Thread 0xffff6866ef90 (LWP 781) "acceptor-781" 0x0000ffffb0b50780 in accept4 () from /lib64/libc.so.6
> 60 Thread 0xffff67e5ef90 (LWP 782) "sq_acceptor" 0x0000ffffb0b44e24 in poll () from /lib64/libc.so.6
> 59 Thread 0xffff6764ef90 (LWP 783) "heartbeat-783" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 58 Thread 0xffff66e3ef90 (LWP 784) "heartbeat-784" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 57 Thread 0xffff6662ef90 (LWP 785) "heartbeat-785" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 56 Thread 0xffff65e1ef90 (LWP 786) "maintenance_sch" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 55 Thread 0xffff51b9ef90 (LWP 1000) "sq_worker" 0x0000ffffb0b44e24 in poll () from /lib64/libc.so.6
> 54 Thread 0xffff9fb4ef90 (LWP 10412) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 53 Thread 0xffff5b4cef90 (LWP 10567) "raft [worker]-1" 0x0000ffffb12fefcc in __lll_lock_wait () from /lib64/libpthread.so.0
> 52 Thread 0xffff4e32ef90 (LWP 10592) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 51 Thread 0xffff5d50ef90 (LWP 10599) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 50 Thread 0xffff4db1ef90 (LWP 10610) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 49 Thread 0xffff4d30ef90 (LWP 10609) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 48 Thread 0xffff4fb5ef90 (LWP 10611) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 47 Thread 0xffff4cafef90 (LWP 10612) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 46 Thread 0xffffa6c7ef90 (LWP 10613) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 45 Thread 0xffffa646ef90 (LWP 10614) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 44 Thread 0xffffa5c5ef90 (LWP 10615) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 43 Thread 0xffffa544ef90 (LWP 10616) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 42 Thread 0xffffa4c3ef90 (LWP 10617) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 41 Thread 0xffffa442ef90 (LWP 10618) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 40 Thread 0xffffa3c1ef90 (LWP 10619) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 39 Thread 0xffffa340ef90 (LWP 10620) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 38 Thread 0xffffa2bfef90 (LWP 10621) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 37 Thread 0xffffa23eef90 (LWP 10622) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 36 Thread 0xffffa1bdef90 (LWP 10623) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 35 Thread 0xffffa0b6ef90 (LWP 10624) "raft [worker]-1" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 34 Thread 0xffffa035ef90 (LWP 10625) "raft [worker]-1" 0x00000000020c7d84 in store (__m=std::memory_order_relaxed, __i=40, this=0x27a358f74) at /usr/include/c++/4.8.2/bits/atomic_base.h:474
> 33 Thread 0xffff9f33ef90 (LWP 10626) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 32 Thread 0xffff9743ef90 (LWP 10627) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 31 Thread 0xffff6560ef90 (LWP 10628) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 30 Thread 0xffff64dfef90 (LWP 10629) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 29 Thread 0xffff645eef90 (LWP 10630) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 28 Thread 0xffff63ddef90 (LWP 10631) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 27 Thread 0xffff635cef90 (LWP 10632) "rpc worker-735" 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff635cab74, p=<optimized out>)
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
> 26 Thread 0xffff62dbef90 (LWP 10633) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 25 Thread 0xffff625aef90 (LWP 10634) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 24 Thread 0xffff61d9ef90 (LWP 10635) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 23 Thread 0xffff6158ef90 (LWP 10636) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 22 Thread 0xffff60d7ef90 (LWP 10637) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 21 Thread 0xffff6056ef90 (LWP 10638) "prepare [worker" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 20 Thread 0xffff5fd5ef90 (LWP 10639) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 19 Thread 0xffff5f54ef90 (LWP 10640) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 18 Thread 0xffff5ed3ef90 (LWP 10641) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 17 Thread 0xffff5e52ef90 (LWP 10642) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 16 Thread 0xffff5dd1ef90 (LWP 10643) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 15 Thread 0xffff5ccfef90 (LWP 10644) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 14 Thread 0xffff5c4eef90 (LWP 10645) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 13 Thread 0xffff5bcdef90 (LWP 10646) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 12 Thread 0xffff5acbef90 (LWP 10647) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 11 Thread 0xffff5a4aef90 (LWP 10648) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 10 Thread 0xffff59c9ef90 (LWP 10649) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 9 Thread 0xffff5948ef90 (LWP 10650) "apply [worker]-" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 8 Thread 0xffff58c7ef90 (LWP 10651) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 7 Thread 0xffff5846ef90 (LWP 10652) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 6 Thread 0xffff57c5ef90 (LWP 10653) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 5 Thread 0xffff5744ef90 (LWP 10654) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 4 Thread 0xffff56c3ef90 (LWP 10655) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 3 Thread 0xffff5642ef90 (LWP 10656) "wal-append [wor" 0x0000ffffb12fc41c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
> 2 Thread 0xffff55c1ef90 (LWP 10657) "wal-append [wor" 0x00000000020bbffc in kudu::consensus::OpId::~OpId (this=0x1f8b90198, __in_chrg=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/build/release/src/kudu/consensus/opid.pb.cc:126
> * 1 Thread 0xffffb142b450 (LWP 7) "kudu-tserver" 0x0000ffffb12ffcb4 in nanosleep () from /lib64/libpthread.so.0
> ...
> Thread 27 (Thread 0xffff635cef90 (LWP 10632)):
> #0 0x000000000208358c in sys_wait4 (o=1073741824, r=0x0, s=0xffff635cab74, p=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3692
> #1 sys_waitpid (options=1073741824, status=0xffff635cab74, pid=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/third_party/lss/linux_syscall_support.h:3697
> #2 google_breakpad::ExceptionHandler::GenerateDump (this=0x12d0c4d0, context=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:553
> #3 0x00000000020838fc in google_breakpad::ExceptionHandler::SignalHandler (sig=11, info=0xffff635cac90, uc=0xffff635cad10)
> at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/breakpad-9eac2058b70615519b2c4d8c6bdbfca1bd079e39/src/client/linux/handler/exception_handler.cc:376
> #4 <signal handler called>
> #5 dwarf_get (c=0xffff635cce20, val=<synthetic pointer>, loc=...) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/include/tdep-aarch64/libunwind_i.h:146
> #6 apply_reg_state (c=c@entry=0xffff635cce20, rs=rs@entry=0xffff635cc718) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:878
> #7 0x0000000002065dd8 in _ULaarch64_dwarf_step (c=c@entry=0xffff635cce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/dwarf/Gparser.c:967
> #8 0x0000000002063ef4 in _ULaarch64_step (cursor=0xffff635cce20) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/thirdparty/src/libunwind-1.4.0/src/aarch64/Gstep.c:146
> #9 0x0000000001f88fc0 in kudu::StackTrace::Collect (this=this@entry=0x75d33f18, skip_frames=0, skip_frames@entry=1) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:612
> #10 0x0000000001f8948c in Collect (skip_frames=1, this=0x75d33f18) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:288
> #11 kudu::(anonymous namespace)::HandleStackTraceSignal (info=<optimized out>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/debug-util.cc:286
> #12 <signal handler called>
> #13 0x0000ffffb0b4f640 in clone () from /lib64/libc.so.6
> #14 0x0000ffffb12f6d78 in do_clone.constprop.3 () from /lib64/libpthread.so.0
> #15 0x0000ffffb12f84f8 in pthread_create@@GLIBC_2.17 () from /lib64/libpthread.so.0
> #16 0x00000000020268bc in kudu::Thread::StartThread(std::string, std::string, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) (category=..., name=..., functor=..., flags=<error reading variable: Cannot access memory at address 0x69>,
> holder=<error reading variable: Cannot access memory at address 0x89>) at /opt/hbp3.0_arm/packages/kudu/apache-kudu-1.13.0/src/kudu/util/thread.cc:618
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> {quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)