You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/11/30 18:31:00 UTC
[jira] [Commented] (KUDU-3108) Tablet server crashes when handle
diffscan request
[ https://issues.apache.org/jira/browse/KUDU-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240961#comment-17240961 ]
ASF subversion and git services commented on KUDU-3108:
-------------------------------------------------------
Commit 6f807b136dfc072b019bdb4a5f1719603096898f in kudu's branch refs/heads/master from Andrew Wong
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=6f807b1 ]
KUDU-3108: fix invalid memory accesses in merge iterator
The 'hotmaxes_' heap and 'hot_' heap in the MergeIterator are meant to
represent the same set of iterator states currently deemed "hot" in the
optimal merge algorithm[1]. They used different ordering constraints to
allow constant time access to the smallest last row and the smallest
next row across all hot states respectively. However, we were using
pop() on both heaps when iterator states were no longer deemed hot,
incorrectly expecting that the call would remove the same iterator state
from both heaps.
In the case that this pop() was followed by the destruction of the
sub-iterator (e.g. if the iterator was fully exhausted), this would
leave an iterator state in the heaps that pointed at destructed state:
F1030 21:16:58.411253 40800 schema.h:706] Check failed: KeyEquals(*lhs.schema()) && KeyEquals(*rhs.schema())
*** Check failure stack trace: ***
*** Aborted at 1604117818 (unix time) try "date -d @1604117818" if you are using GNU date ***
PC: @ 0x7f701fcf11d7 __GI_raise
*** SIGABRT (@0x111700009efd) received by PID 40701 (TID 0x7f6ff0f47700) from PID 40701; stack trace: ***
@ 0x7f7026a70370 (unknown)
@ 0x7f701fcf11d7 __GI_raise
@ 0x7f701fcf28c8 __GI_abort
@ 0x7f70224377b9 google::logging_fail()
@ 0x7f7022438f8d google::LogMessage::Fail()
@ 0x7f702243aee3 google::LogMessage::SendToLog()
@ 0x7f7022438ae9 google::LogMessage::Flush()
@ 0x7f702243b86f google::LogMessageFatal::~LogMessageFatal()
@ 0x7f702cc99fbc kudu::Schema::Compare<>()
@ 0x7f7026167cfd kudu::MergeIterator::RefillHotHeap()
@ 0x7f7026167357 kudu::MergeIterator::AdvanceAndReheap()
@ 0x7f7026169617 kudu::MergeIterator::MaterializeOneRow()
@ 0x7f70261688e9 kudu::MergeIterator::NextBlock()
@ 0x7f702cbddd9b kudu::tablet::Tablet::Iterator::NextBlock()
@ 0x7f70317bcab3 kudu::tserver::TabletServiceImpl::HandleContinueScanRequest()
@ 0x7f70317bb857 kudu::tserver::TabletServiceImpl::HandleNewScanRequest()
@ 0x7f70317b464e kudu::tserver::TabletServiceImpl::Scan()
@ 0x7f702ddfd762 _ZZN4kudu7tserver21TabletServerServiceIfC1ERK13scoped_refptrINS_12MetricEntityEERKS2_INS_3rpc13ResultTrackerEEENKUlPKN6google8protobuf7MessageEPSE_PNS7_10RpcContextEE4_clESG_SH_SJ_
@ 0x7f702de0064d _ZNSt17_Function_handlerIFvPKN6google8protobuf7MessageEPS2_PN4kudu3rpc10RpcContextEEZNS6_7tserver21TabletServerServiceIfC1ERK13scoped_refptrINS6_12MetricEntityEERKSD_INS7_13ResultTrackerEEEUlS4_S5_S9_E4_E9_M_invokeERKSt9_Any_dataS4_S5_S9_
@ 0x7f702b4ddcc2 std::function<>::operator()()
@ 0x7f702b4dd6ed kudu::rpc::GeneratedServiceIf::Handle()
@ 0x7f702b4dfff8 kudu::rpc::ServicePool::RunThread()
@ 0x7f702b4de8c5 _ZZN4kudu3rpc11ServicePool4InitEiENKUlvE_clEv
@ 0x7f702b4e0337 _ZNSt17_Function_handlerIFvvEZN4kudu3rpc11ServicePool4InitEiEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x7f7033524b9c std::function<>::operator()()
@ 0x7f70248227e0 kudu::Thread::SuperviseThread()
@ 0x7f7026a68dc5 start_thread
@ 0x7f701fdb376d __clone
Aborted
This patch removes the 'hotmaxes_' min heap in favor of a two-heap
variant of the merge algorithm that Adar described to me that uses the
last value in the top iterator in the hot heap. This is not as optimal
in terms of merge window size, but is still correct and avoids this bug.
I experimented with some other approaches, described below. I ran the
same test used in 1567dec086 (generic_iterators-test TestMerge and
TestMergeNonOverlapping with 10000 rows per list), averaged over five
runs:
a: An iteration of this patch that used std::set for hotmaxes, a call
to find() before calling Advance(), and a position-based erase()
after. The find() allowed for an erase() call that did not rely at
all on the comparator, which may have pointed at destructed state
following the call to Advance().
b: An iteration of this patch that used std::set for hotmaxes, a call
to a value-based erase() before calling Advance(). The call to
erase() before Advance() ensured Advance() calls never interfered
with our ability to compare and erase from hotmaxes, at the
potential cost of an extra insert if the iterator were still hot.
c: The original version that uses heap::pop() after calling Advance().
d: This patch, that doesn't use hotmaxes, and instead uses the last row
of the top value in the hot heap to define the upper bound of the
merge window.
Parameters | a | b | c | d
----------------------------+----------+----------+----------+---------
overlapping, 10 lists | 0.059s | 0.0744s | 0.0472s | 0.0478s
overlapping, 100 lists | 0.6726s | 0.8876s | 0.4938s | 0.491s
overlapping, 1000 lists | 15.5588s | 18.87s | 10.3554s | 10.157s
non-overlapping, 10 lists | 0.011s | 0.0114s | 0.0106s | 0.0092s
non-overlapping, 100 lists | 0.0786s | 0.0794s | 0.083s | 0.0682s
non-overlapping, 1000 lists | 0.7824s | 0.7346s | 0.7174s | 0.6884s
I also ran an ordered scan with `kudu perf tablet_scan` on a 65GiB
tablet hosted on a single disk with 1667 rowsets and an average rowset
height of five, averaged over six runs each (I omitted testing version b
since it was the worst-performing of the above):
Results | a | c | d
----------+----------+----------+----------
real time | 1070.69s | 1166.96s | 1048.57s
stdev | 19.48s | 25.14s | 19.54s
I didn't profile these runs in depth, but the measurements suggest that
the maintenance of the hotmaxes set may add overhead that isn't always
recouped by an optimally-sized merge window. I left a TODO to experiment
further with hotmaxes of different data structures (e.g. absl::btree
seems like a good candidate).
To exercise these codepaths more rigorously, I bumped fuzz-itest's
default keyspace size to 5. Some bugs (this one included) can only be
created when there are a mix of overlapping and non-overlapping rowsets,
which is impossible to achieve with the current keyspace size of 2.
[1] https://docs.google.com/document/d/1uP0ubjM6ulnKVCRrXtwT_dqrTWjF9tlFSRk0JN2e_O0/edit#
Change-Id: I8ec1cd3fd67ec4ea92a55b5b0ce555123748824d
Reviewed-on: http://gerrit.cloudera.org:8080/16777
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
> Tablet server crashes when handle diffscan request
> ---------------------------------------------------
>
> Key: KUDU-3108
> URL: https://issues.apache.org/jira/browse/KUDU-3108
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 1.10.1
> Reporter: YifanZhang
> Priority: Major
>
> When we did an incremental backup for tables in a cluster with 20 tservers, 3 tservers crashed, coredump stacks are the same:
> {code:java}
> Unable to find source-code formatter for language: shell. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yamlProgram terminated with signal 11, Segmentation fault.Program terminated with signal 11, Segmentation fault.
> #0 kudu::Schema::Compare<kudu::RowBlockRow, kudu::RowBlockRow> (this=0x25b883680, lhs=..., rhs=...) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267
> 267 /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h: No such file or directory.
> Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-gssapi-2.1.26-20.el7_2.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 cyrus-sasl-md5-2.1.26-20.el7_2.x86_64 cyrus-sasl-plain-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libdb-5.3.21-19.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-6.el7.x86_64 ncurses-libs-5.9-13.20130511.el7.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-30.el7_3.8.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) bt
> #0 kudu::Schema::Compare<kudu::RowBlockRow, kudu::RowBlockRow> (this=0x25b883680, lhs=..., rhs=...) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267
> #1 0x0000000001da51fb in kudu::MergeIterator::RefillHotHeap (this=this@entry=0x78f6ec500) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:720
> #2 0x0000000001da622b in kudu::MergeIterator::AdvanceAndReheap (this=this@entry=0x78f6ec500, state=0xd1661a000, num_rows_to_advance=num_rows_to_advance@entry=1) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:690
> #3 0x0000000001da7927 in kudu::MergeIterator::MaterializeOneRow (this=this@entry=0x78f6ec500, dst=dst@entry=0x7f0d5cc9ffc0, dst_row_idx=dst_row_idx@entry=0x7f0d5cc9fbb0) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:894
> #4 0x0000000001da7de3 in kudu::MergeIterator::NextBlock (this=0x78f6ec500, dst=0x7f0d5cc9ffc0) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:796
> #5 0x0000000000a9ff19 in kudu::tablet::Tablet::Iterator::NextBlock (this=<optimized out>, dst=<optimized out>) at /home/zhangyifan8/work/kudu-xm/src/kudu/tablet/tablet.cc:2499
> #6 0x000000000095475c in kudu::tserver::TabletServiceImpl::HandleContinueScanRequest (this=this@entry=0x53b5a90, req=req@entry=0x7f0d5cca0720, rpc_context=rpc_context@entry=0x5e512a460, result_collector=result_collector@entry=0x7f0d5cca0a00, has_more_results=has_more_results@entry=0x7f0d5cca0886, error_code=error_code@entry=0x7f0d5cca0888) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2565
> #7 0x0000000000966564 in kudu::tserver::TabletServiceImpl::HandleNewScanRequest (this=this@entry=0x53b5a90, replica=0xf5c0189c0, req=req@entry=0x2a15c240, rpc_context=rpc_context@entry=0x5e512a460, result_collector=result_collector@entry=0x7f0d5cca0a00, scanner_id=scanner_id@entry=0x7f0d5cca0940, snap_timestamp=snap_timestamp@entry=0x7f0d5cca0950, has_more_results=has_more_results@entry=0x7f0d5cca0886, error_code=error_code@entry=0x7f0d5cca0888) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2476
> #8 0x0000000000967f4b in kudu::tserver::TabletServiceImpl::Scan (this=0x53b5a90, req=0x2a15c240, resp=0x56f9be6c0, context=0x5e512a460) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:1674
> #9 0x0000000001d2e449 in operator() (__args#2=0x5e512a460, __args#1=0x56f9be6c0, __args#0=<optimized out>, this=0x497ecdd8) at /usr/include/c++/4.8.2/functional:2471
> #10 kudu::rpc::GeneratedServiceIf::Handle (this=0x53b5a90, call=<optimized out>) at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_if.cc:139
> #11 0x0000000001d2eb49 in kudu::rpc::ServicePool::RunThread (this=0x2ab69560) at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_pool.cc:225
> #12 0x0000000001e9e924 in operator() (this=0x90fb52e8) at /home/zhangyifan8/work/kudu-xm/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771
> #13 kudu::Thread::SuperviseThread (arg=0x90fb52c0) at /home/zhangyifan8/work/kudu-xm/src/kudu/util/thread.cc:657
> #14 0x00007f103b20cdc5 in start_thread () from /lib64/libpthread.so.0
> #15 0x00007f103956673d in clone () from /lib64/libc.so.6
> {code}
> Before we did first time full backup, we set extra config for these tables via CLI tool:
> {code:java}
> kudu table set_extra_config @c3tst-master xxx kudu.table.history_max_age_sec 604800
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)