You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Will Berkeley (JIRA)" <ji...@apache.org> on 2018/01/30 19:01:00 UTC
[jira] [Created] (KUDU-2275) SIGSEGV due to bug in libunwind
Will Berkeley created KUDU-2275:
-----------------------------------
Summary: SIGSEGV due to bug in libunwind
Key: KUDU-2275
URL: https://issues.apache.org/jira/browse/KUDU-2275
Project: Kudu
Issue Type: Bug
Affects Versions: 1.6.0
Reporter: Will Berkeley
Rarely, the kernel stack watchdog can cause a segfault due to a bug in libunwind.
{noformat}
*** Aborted at 1516180006 (unix time) try "date -d @1516180006" if you are using GNU date ***
PC: @ 0x8c94b4 (unknown)
*** SIGSEGV (@0x7f27173e0000) received by PID 22279 (TID 0x7f270f87f700) from PID 389939200; stack trace: ***{noformat}
From a core file (produced from the minidump), the backtrace is
{noformat}
#0 access_mem (as=<optimized out>, addr=139805870391296, val=0x7f270f87bcc0, write=<optimized out>, arg=<optimized out>)
at /usr/src/debug/kudu-1.5.0-cdh5.13.1/thirdparty/src/libunwind-1.1a/src/x86_64/Ginit.c:173
#1 0x00000000008c8e02 in is_plt_entry (c=0x7f270f87c0e0) at /usr/src/debug/kudu-1.5.0-cdh5.13.1/thirdparty/src/libunwind-1.1a/src/x86_64/Gstep.c:43
#2 _ULx86_64_step (cursor=0x7f270f87c0e0) at /usr/src/debug/kudu-1.5.0-cdh5.13.1/thirdparty/src/libunwind-1.1a/src/x86_64/Gstep.c:125
#3 0x00000000008c412d in google::GetStackTrace (result=result@entry=0x292c0c8, max_depth=max_depth@entry=16, skip_count=0, skip_count@entry=2)
at /usr/src/debug/kudu-1.5.0-cdh5.13.1/thirdparty/src/glog-0.3.5/src/stacktrace_libunwind-inl.h:78
#4 0x0000000001a9be8c in Collect (skip_frames=2, this=0x292c0c0) at /usr/src/debug/kudu-1.5.0-cdh5.13.1/src/kudu/util/debug-util.cc:350
#5 kudu::(anonymous namespace)::HandleStackTraceSignal (signum=<optimized out>) at /usr/src/debug/kudu-1.5.0-cdh5.13.1/src/kudu/util/debug-util.cc:176
#6 0x00007f2716854670 in _quicksort () from ./lib64/libc.so.6
#7 0x0000000000000000 in ?? (){noformat}
Note that addr = 139805870391296 = 0x7f27173e0000.
The segfault happens because libunwind is accessing invalid memory it's supposed to have validated:
{code:java}
/* validate address */
const struct cursor *c = (const struct cursor *)arg;
if (likely (c != NULL) && unlikely (c->validate)
&& unlikely (validate_mem (addr)))
return -1;
*val = *(unw_word_t *) addr;{code}
[Others|https://lists.nongnu.org/archive/html/libunwind-devel/2016-09/msg00001.html] have seen this same problem before.
There's also a fix for this issue in commit 836c91c43d7a996028aa7e8d1f53630a6b8e7cbe. It's not in any release of libunwind yet, so we could do one of the following
# upgrade libunwind to 1.2 (most recent release) and patch in the fix
# upgrade to a snapshot containing the fix
To workaround, one can set --hung_task_check_interval_ms to a large value like 2^30, so the stack watchdog runs very rarely (although the flag is a 32-bit signed integer, so not too big).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)