You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/03/31 20:17:00 UTC
[jira] [Commented] (IMPALA-9568) Template tuples are initialized
multiple times
[ https://issues.apache.org/jira/browse/IMPALA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072130#comment-17072130 ]
ASF subversion and git services commented on IMPALA-9568:
---------------------------------------------------------
Commit 0038487267ccba77e6cb17a582039d6b35ff947c in impala's branch refs/heads/master from Sahil Takiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0038487 ]
IMPALA-9568: Template tuples are initialized multiple times
Template tuples are initialized multiple times, concurrently, which
leads to a data race reported by TSAN. The template tuples should just
be initialized once. This patch removes the template tuple
initialization done in the HDFS scanners and just copies them from the
HdfsScanNodeBase.
After this patch, data-load is now TSAN clean. I removed the TSAN flag
'halt_on_error=0' so that TSAN now crashes the process if a TSAN bug is
discovered. This will allow us to setup a Jenkins job that runs dataload
+ be/ tests, and will crash if any new TSAN bugs are discovered.
Testing:
* Ran exhaustive tests
Change-Id: I3bd3554b2b919b117a0c9ae86dfc0a75ae4129d2
Reviewed-on: http://gerrit.cloudera.org:8080/15604
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Template tuples are initialized multiple times
> ----------------------------------------------
>
> Key: IMPALA-9568
> URL: https://issues.apache.org/jira/browse/IMPALA-9568
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> TSAN is reporting the following data race with template tuple initialization:
> {code:java}
> WARNING: ThreadSanitizer: data race (pid=8124)
> Write of size 4 at 0x7b30001692f4 by thread T355:
> #0 impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, impala::TupleRow const*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:279:23 (impalad+0x2553330)
> #1 impala::ScalarExprEvaluator::GetValue(impala::TupleRow const*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:253:10 (impalad+0x255357d)
> #2 impala::HdfsScanNodeBase::InitTemplateTuple(std::vector<impala::ScalarExprEvaluator*, std::allocator<impala::ScalarExprEvaluator*> > const&, impala::MemPool*, impala::RuntimeState*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:900:27 (impalad+0x237c74a)
> #3 impala::HdfsScanner::Open(impala::ScannerContext*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scanner.cc:110:33 (impalad+0x23a6437)
> #4 impala::HdfsTextScanner::Open(impala::ScannerContext*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:780:3 (impalad+0x23e6121)
> #5 impala::HdfsScanNodeBase::CreateAndOpenScannerHelper(impala::HdfsPartitionDescriptor*, impala::ScannerContext*, boost::scoped_ptr<impala::HdfsScanner>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:887:3 (impalad+0x237fc76)
> #6 impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::io::ScanRange*, long*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:479:19 (impalad+0x24daeba)
> #7 impala::HdfsScanNode::ScannerThread(bool, long) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:417:7 (impalad+0x24da84e)
> #8 impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0::operator()() const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:338:13 (impalad+0x24dbd06)
> #9 boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0, void>::invoke(boost::detail::function::function_buffer&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x24dbb19)
> #10 boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x1d0af41)
> #11 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 (impalad+0x22c3c66)
> #12 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 (impalad+0x22cbe2c)
> #13 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x22cbd43)
> #14 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 (impalad+0x22cba30)
> #15 thread_proxy <null> (impalad+0x2fb6509)
> Previous write of size 4 at 0x7b30001692f4 by thread T337:
> #0 impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, impala::TupleRow const*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:279:23 (impalad+0x2553330)
> #1 impala::ScalarExprEvaluator::GetValue(impala::TupleRow const*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:253:10 (impalad+0x255357d)
> #2 impala::HdfsScanNodeBase::InitTemplateTuple(std::vector<impala::ScalarExprEvaluator*, std::allocator<impala::ScalarExprEvaluator*> > const&, impala::MemPool*, impala::RuntimeState*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:900:27 (impalad+0x237c74a)
> #3 impala::HdfsScanner::Open(impala::ScannerContext*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scanner.cc:110:33 (impalad+0x23a6437)
> #4 impala::HdfsTextScanner::Open(impala::ScannerContext*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:780:3 (impalad+0x23e6121)
> #5 impala::HdfsScanNodeBase::CreateAndOpenScannerHelper(impala::HdfsPartitionDescriptor*, impala::ScannerContext*, boost::scoped_ptr<impala::HdfsScanner>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:887:3 (impalad+0x237fc76)
> #6 impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::io::ScanRange*, long*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:479:19 (impalad+0x24daeba)
> #7 impala::HdfsScanNode::ScannerThread(bool, long) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:417:7 (impalad+0x24da84e)
> #8 impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0::operator()() const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:338:13 (impalad+0x24dbd06)
> #9 boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0, void>::invoke(boost::detail::function::function_buffer&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x24dbb19)
> #10 boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x1d0af41)
> #11 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 (impalad+0x22c3c66)
> #12 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 (impalad+0x22cbe2c)
> #13 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x22cbd43)
> #14 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 (impalad+0x22cba30)
> #15 thread_proxy <null> (impalad+0x2fb6509)
> Location is heap block of size 192 at 0x7b30001692c0 allocated by thread T349:
> #0 operator new(unsigned long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:44 (impalad+0x192f603)
> #1 impala::ScalarExprEvaluator::Create(impala::ScalarExpr const&, impala::RuntimeState*, impala::ObjectPool*, impala::MemPool*, impala::MemPool*, impala::ScalarExprEvaluator**) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:79:21 (impalad+0x25516b9)
> #2 impala::ScalarExprEvaluator::Create(std::vector<impala::ScalarExpr*, std::allocator<impala::ScalarExpr*> > const&, impala::RuntimeState*, impala::ObjectPool*, impala::MemPool*, impala::MemPool*, std::vector<impala::ScalarExprEvaluator*, std::allocator<impala::ScalarExprEvaluator*> >*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:101:21 (impalad+0x2551f30)
> #3 impala::DescriptorTbl::CreatePartKeyExprs(impala::HdfsTableDescriptor const&, impala::ObjectPool*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:516:5 (impalad+0x1f2323e)
> #4 impala::DescriptorTbl::CreateTblDescriptorInternal(impala::TTableDescriptor const&, impala::ObjectPool*, impala::TableDescriptor**) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:557:7 (impalad+0x1f238c2)
> #5 impala::DescriptorTbl::CreateInternal(impala::ObjectPool*, impala::TDescriptorTable const&, impala::DescriptorTbl**) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:588:5 (impalad+0x1f23bce)
> #6 impala::DescriptorTbl::Create(impala::ObjectPool*, impala::TDescriptorTableSerialized const&, impala::DescriptorTbl**) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:579:10 (impalad+0x1f23aa5)
> #7 impala::QueryState::StartFInstances() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:570:7 (impalad+0x1f4af0e)
> #8 impala::QueryExecMgr::ExecuteQueryHelper(impala::QueryState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-exec-mgr.cc:141:7 (impalad+0x1f96bc9)
> #9 boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>::operator()(impala::QueryExecMgr*, impala::QueryState*) const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:165:29 (impalad+0x1f97ebd)
> #10 void boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::QueryState*> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>&, boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:319:9 (impalad+0x1f97dfd)
> #11 boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::QueryState*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x1f97d63)
> #12 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::QueryState*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x1f97b01)
> #13 boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x1d0af41)
> #14 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 (impalad+0x22c3c66)
> #15 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 (impalad+0x22cbe2c)
> #16 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x22cbd43)
> #17 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 (impalad+0x22cba30)
> #18 thread_proxy <null> (impalad+0x2fb6509)
> Thread T355 (tid=3630, running) created by thread T364 at:
> #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (impalad+0x18c6feb)
> #1 boost::thread::start_thread_noexcept() <null> (impalad+0x2fb5899)
> #2 boost::thread::thread<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13 (impalad+0x22c643b)
> #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:317:13 (impalad+0x22c3605)
> #4 impala::Status impala::Thread::Create<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0>(std::string const&, std::string const&, impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0 const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.h:74:12 (impalad+0x24da3ad)
> #5 impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:341:14 (impalad+0x24d99e9)
> #6 impala::HdfsScanNode::AddDiskIoRanges(std::vector<impala::io::ScanRange*, std::allocator<impala::io::ScanRange*> > const&, impala::EnqueueLocation) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:213:24 (impalad+0x24d9ffe)
> #7 impala::HdfsScanNodeBase::AddDiskIoRanges(impala::HdfsFileDesc const*, impala::EnqueueLocation) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.h:356:12 (impalad+0x23e681b)
> #8 impala::HdfsTextScanner::IssueInitialRanges(impala::HdfsScanNodeBase*, std::vector<impala::HdfsFileDesc*, std::allocator<impala::HdfsFileDesc*> > const&) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:86:9 (impalad+0x23df95b)
> #9 impala::HdfsScanNodeBase::IssueInitialScanRanges(impala::RuntimeState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:639:9 (impalad+0x237dce9)
> #10 impala::HdfsScanNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:91:21 (impalad+0x24d8b5c)
> #11 impala::FragmentInstanceState::ExecInternal() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:383:7 (impalad+0x1f3b9e5)
> #12 impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:98:14 (impalad+0x1f398a6)
> #13 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:697:24 (impalad+0x1f4c113)
> #14 impala::QueryState::StartFInstances()::$_7::operator()() const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:606:35 (impalad+0x1f4e2d2)
> #15 boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_7, void>::invoke(boost::detail::function::function_buffer&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x1f4e0f9)
> #16 boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x1d0af41)
> #17 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 (impalad+0x22c3c66)
> #18 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 (impalad+0x22cbe2c)
> #19 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x22cbd43)
> #20 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 (impalad+0x22cba30)
> #21 thread_proxy <null> (impalad+0x2fb6509)
> Thread T337 (tid=3629, running) created by thread T364 at:
> #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (impalad+0x18c6feb)
> #1 boost::thread::start_thread_noexcept() <null> (impalad+0x2fb5899)
> #2 boost::thread::thread<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13 (impalad+0x22c643b)
> #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:317:13 (impalad+0x22c3605)
> #4 impala::Status impala::Thread::Create<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0>(std::string const&, std::string const&, impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0 const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.h:74:12 (impalad+0x24da3ad)
> #5 impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:341:14 (impalad+0x24d99e9)
> #6 impala::HdfsScanNode::AddDiskIoRanges(std::vector<impala::io::ScanRange*, std::allocator<impala::io::ScanRange*> > const&, impala::EnqueueLocation) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:213:24 (impalad+0x24d9ffe)
> #7 impala::HdfsScanNodeBase::AddDiskIoRanges(impala::HdfsFileDesc const*, impala::EnqueueLocation) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.h:356:12 (impalad+0x23e681b)
> #8 impala::HdfsTextScanner::IssueInitialRanges(impala::HdfsScanNodeBase*, std::vector<impala::HdfsFileDesc*, std::allocator<impala::HdfsFileDesc*> > const&) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:86:9 (impalad+0x23df95b)
> #9 impala::HdfsScanNodeBase::IssueInitialScanRanges(impala::RuntimeState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:639:9 (impalad+0x237dce9)
> #10 impala::HdfsScanNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:91:21 (impalad+0x24d8b5c)
> #11 impala::FragmentInstanceState::ExecInternal() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:383:7 (impalad+0x1f3b9e5)
> #12 impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:98:14 (impalad+0x1f398a6)
> #13 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:697:24 (impalad+0x1f4c113)
> #14 impala::QueryState::StartFInstances()::$_7::operator()() const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:606:35 (impalad+0x1f4e2d2)
> #15 boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_7, void>::invoke(boost::detail::function::function_buffer&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x1f4e0f9)
> #16 boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x1d0af41)
> #17 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 (impalad+0x22c3c66)
> #18 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 (impalad+0x22cbe2c)
> #19 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x22cbd43)
> #20 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 (impalad+0x22cba30)
> #21 thread_proxy <null> (impalad+0x2fb6509)
> Thread T349 (tid=3561, running) created by thread T97 at:
> #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (impalad+0x18c6feb)
> #1 boost::thread::start_thread_noexcept() <null> (impalad+0x2fb5899)
> #2 boost::thread::thread<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13 (impalad+0x22c643b)
> #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:317:13 (impalad+0x22c3605)
> #4 impala::Status impala::Thread::Create<void (impala::QueryExecMgr::*)(impala::QueryState*), impala::QueryExecMgr*, impala::QueryState*>(std::string const&, std::string const&, void (impala::QueryExecMgr::* const&)(impala::QueryState*), impala::QueryExecMgr* const&, impala::QueryState* const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.h:89:12 (impalad+0x1f971a7)
> #5 impala::QueryExecMgr::StartQuery(impala::ExecQueryFInstancesRequestPB const*, impala::TQueryCtx const&, impala::TExecPlanFragmentInfo const&) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-exec-mgr.cc:69:12 (impalad+0x1f96092)
> #6 impala::ControlService::ExecQueryFInstances(impala::ExecQueryFInstancesRequestPB const*, impala::ExecQueryFInstancesResponsePB*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/control-service.cc:156:66 (impalad+0x2065dee)
> #7 impala::ControlServiceIf::ControlServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/generated-sources/gen-cpp/control_service.service.cc:70:13 (impalad+0x2113024)
> #8 std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::ControlServiceIf::ControlServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2 (impalad+0x2112e0e)
> #9 std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14 (impalad+0x27afa68)
> #10 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3 (impalad+0x27af3ce)
> #11 impala::ImpalaServicePool::RunThread() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15 (impalad+0x1e3260f)
> #12 boost::_mfi::mf0<void, impala::ImpalaServicePool>::operator()(impala::ImpalaServicePool*) const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:49:29 (impalad+0x1e38b76)
> #13 void boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> >::operator()<boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, impala::ImpalaServicePool>&, boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:259:9 (impalad+0x1e38aca)
> #14 boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x1e38a53)
> #15 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >, void>::invoke(boost::detail::function::function_buffer&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x1e38849)
> #16 boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x1d0af41)
> #17 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 (impalad+0x22c3c66)
> #18 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 (impalad+0x22cbe2c)
> #19 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x22cbd43)
> #20 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 (impalad+0x22cba30)
> #21 thread_proxy <null> (impalad+0x2fb6509)SUMMARY: ThreadSanitizer: data race /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:279:23 in impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, impala::TupleRow const*) {code}
> The odd thing is that it looks like template tuples are initialized multiple times, once in {{HdfsScanNodeBase::Open}} --> {{HdfsScanNodeBase::InitTemplateTuple}} and again in {{HdfsScanner::Open}}. I think a simple fix here would be for the scanners to not initialize the template tuples and just use the one initialized in {{HdfsScanNodeBase}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org