You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Alexey Serbin (JIRA)" <ji...@apache.org> on 2019/08/17 00:17:00 UTC
[jira] [Commented] (KUDU-2212) TSAN "destroy of locked mutex"
failure in kudu-admin-test
[ https://issues.apache.org/jira/browse/KUDU-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909500#comment-16909500 ]
Alexey Serbin commented on KUDU-2212:
-------------------------------------
Another example of the TSAN warning message about destruction of a locked mutex when {{kudu}} CLI exits:
{noformat}
==================
WARNING: ThreadSanitizer: destroy of a locked mutex (pid=2813)
#0 pthread_rwlock_destroy /data/8/aserbin/Projects/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1313 (kudu+0x4bbaf4)
#1 glog_internal_namespace_::Mutex::~Mutex() /data/8/aserbin/Projects/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:249:30 (libglog.so.0+0x16468)
#2 cxa_at_exit_wrapper(void*) /data/8/aserbin/Projects/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (kudu+0x48be83)
and:
#0 pthread_rwlock_wrlock /data/8/aserbin/Projects/kudu/thirdparty/src/llvm-6.0.0.src/projects/co
mpiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1352 (kudu+0x4bb124)
#1 glog_internal_namespace_::Mutex::Lock() /data/8/aserbin/Projects/kudu/thirdparty/src/glog-0.3
.5/src/base/mutex.h:250:30 (libglog.so.0+0x1ab97)
#2 glog_internal_namespace_::MutexLock::MutexLock(glog_internal_namespace_::Mutex*) /data/8/aser
bin/Projects/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:290 (libglog.so.0+0x1ab97)
#3 google::InitVLOG3__(int**, int*, char const*, int) /data/8/aserbin/Projects/kudu/thirdparty/s
rc/glog-0.3.5/src/vlog_is_on.cc:199 (libglog.so.0+0x1ab97)
#4 kudu::rpc::ReactorThread::ShutdownInternal() /data/8/aserbin/Projects/kudu/src/kudu/rpc/react
or.cc:242:3 (libkrpc.so+0xbf553)
#5 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data/8/aserbin/Projects/kudu/src/kud
u/rpc/reactor.cc:322:5 (libkrpc.so+0xc00c4)
#6 void ev::base<ev_async, ev::async>::method_thunk<kudu::rpc::ReactorThread, &kudu::rpc::ReactorThread::AsyncHandler>(ev_loop*, ev_async*, int) /data/8/aserbin/Projects/kudu/thirdparty/installed/
tsan/include/ev++.h:479:7 (libkrpc.so+0xc8d93) #7 ev_invoke_pending /home/aserbin/Projects/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.s
o.4+0x9780)
#8 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/8/aserbin/Projects/kudu/src/kudu/rp
c/reactor.cc:193:3 (libkrpc.so+0xbeea5)
#9 ev_run /home/aserbin/Projects/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa88d)
#10 ev::loop_ref::run(int) /data/8/aserbin/Projects/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc51d8)
#11 kudu::rpc::ReactorThread::RunThread() /data/8/aserbin/Projects/kudu/src/kudu/rpc/reactor.cc:494:9 (libkrpc.so+0xbefea)
#12 boost::_mfi::mf0<void, kudu::rpc::ReactorThread>::operator()(kudu::rpc::ReactorThread*) const /data/8/aserbin/Projects/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc94f9)
#13 void boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >::operator()<boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>&, boost::_bi::list0&, int) /data/8/aserbin/Projects/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc944a)
#14 boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() /data/8/aserbin/Projects/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc93d3)
#15 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/8/aserbin/Projects/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc91c9)
#16 boost::function0<void>::operator()() const /data/8/aserbin/Projects/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb89d1)
#17 kudu::Thread::SuperviseThread(void*) /data/8/aserbin/Projects/kudu/src/kudu/util/thread.cc:657:3 (libkudu_util.so+0x1ec634)
Location is global 'google::vmodule_lock' of size 64 at 0x7f7fa82897e8 (libglog.so.0+0x00000023c7e8)
Mutex M134 (0x7f7fa82897e8) created at:
#0 pthread_rwlock_init /data/8/aserbin/Projects/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1304 (kudu+0x4bbd64)
#1 glog_internal_namespace_::Mutex::Mutex() /data/8/aserbin/Projects/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:247:19 (libglog.so.0+0xcc86)
#2 __cxx_global_var_init.15 /data/8/aserbin/Projects/kudu/thirdparty/src/glog-0.3.5/src/vlog_is_on.cc:120 (libglog.so.0+0xcc86)
#3 _GLOBAL__sub_I_vlog_is_on.cc /data/8/aserbin/Projects/kudu/thirdparty/src/glog-0.3.5/src/vlog_is_on.cc (libglog.so.0+0xcc86)
#4 call_init /build/eglibc-SvCtMH/eglibc-2.19/elf/dl-init.c:78 (ld-linux-x86-64.so.2+0x102d9)
SUMMARY: ThreadSanitizer: destroy of a locked mutex /data/8/aserbin/Projects/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:249:30 in glog_internal_namespace_::Mutex::~Mutex()
==================
ThreadSanitizer: reported 1 warnings
{noformat}
> TSAN "destroy of locked mutex" failure in kudu-admin-test
> ---------------------------------------------------------
>
> Key: KUDU-2212
> URL: https://issues.apache.org/jira/browse/KUDU-2212
> Project: Kudu
> Issue Type: Bug
> Components: test
> Reporter: Todd Lipcon
> Priority: Major
> Attachments: kudu-tool-test.0.txt.gz
>
>
> admin cli test is flaky with:
> WARNING: ThreadSanitizer: destroy of a locked mutex (pid=16401)
> #0 pthread_rwlock_destroy /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1214 (kudu+0x49c866)
> #1 glog_internal_namespace_::Mutex::~Mutex() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:249:30 (libglog.so.0+0x15878)
> #2 at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:375 (kudu+0x4706a3)
> and:
> #0 memcpy /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:655 (kudu+0x49344c)
> #1 std::__1::char_traits<char>::copy(char*, char const*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__string:223:50 (kudu+0x51285e)
> #2 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__init(char const*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/string:1538:5 (libkudu_util.so+0x13c118)
> #3 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(char const*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/string:1547 (libkudu_util.so+0x13c118)
> #4 kudu::flag_tags_internal::FlagTagger::FlagTagger(char const*, char const*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/flag_tags.cc:76 (libkudu_util.so+0x13c118)
> #5 __cxx_global_var_init.8 /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/server/rpc_server.cc:54:1 (libserver_process.so+0x5f7c8)
> #6 _GLOBAL__sub_I_rpc_server.cc /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/server/rpc_server.cc (libserver_process.so+0x5fcfc)
> #7 _dl_init_internal <null> (ld-linux-x86-64.so.2+0xe64e)
> Location is global 'google::log_mutex' of size 64 at 0x7ff79ecd3038 (libglog.so.0+0x00000022b038)
> Mutex M115 (0x7ff79ecd3038) created at:
> #0 pthread_rwlock_init /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1205 (kudu+0x49ca9c)
> #1 glog_internal_namespace_::Mutex::Mutex() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:247:19 (libglog.so.0+0xc5f3)
> #2 __cxx_global_var_init.130 /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/logging.cc:372 (libglog.so.0+0xc5f3)
> #3 _GLOBAL__sub_I_logging.cc /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/logging.cc (libglog.so.0+0xc5f3)
> #4 _dl_init_internal <null> (ld-linux-x86-64.so.2+0xe64e)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)