[jira] [Created] (IMPALA-5135) KRPC : ReportExecStatus RPC can timeout when deserializing large query profiles due to tcmalloc contention

["Mostafa Mokhtar (JIRA)" <ji...@apache.org>]

Mostafa Mokhtar created IMPALA-5135:
---------------------------------------

             Summary: KRPC : ReportExecStatus RPC can timeout when deserializing large query profiles due to tcmalloc contention
                 Key: IMPALA-5135
                 URL: https://issues.apache.org/jira/browse/IMPALA-5135
             Project: IMPALA
          Issue Type: Sub-task
            Reporter: Mostafa Mokhtar
            Assignee: Henry Robinson


Queries with a larger number of fragments can fail with {code}Timed out: ReportExecStatus RPC to 10.17.187.36:22000 timed out after 10.000s (SENT){code}

Vtune shows that while deserializing the query profile the thread can get stuck in tcmalloc

{code}
impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source file]
impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
impalad ! operator new + 0x297 - [unknown source file]
impalad ! __gnu_cxx::new_allocator<impala::TCounter>::allocate + 0x4 - new_allocator.h:104
impalad ! std::vector<impala::TCounter, std::allocator<impala::TCounter>>::resize + 0x7f3 - stl_vector.h:676
impalad ! impala::TRuntimeProfileNode::read + 0xe5a - RuntimeProfile_types.cpp:601
impalad ! impala::TRuntimeProfileTree::read + 0x8ac - RuntimeProfile_types.cpp:982
impalad ! impala::TReportExecStatusParams::read + 0x156 - ImpalaInternalService_types.cpp:2956
impalad ! impala::DeserializeThriftMsg<impala::TReportExecStatusParams> + 0xe5 - thrift-util.h:145
impalad ! impala::DeserializeFromSidecar<impala::TReportExecStatusParams> + 0xb7 - rpc.h:407
impalad ! impala::ExecControlService::ReportExecStatus + 0x21e - impala-internal-service.cc:148
impalad ! std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator() + 0x1c - functional:2439
impalad ! kudu::rpc::GeneratedServiceIf::Handle + 0x188 - service_if.cc:134
impalad ! impala::ImpalaServicePool::RunThread + 0x241 - impala-service-pool.cc:130
impalad ! boost::function0<void>::operator() + 0x1a - function_template.hpp:767
impalad ! impala::Thread::SuperviseThread + 0x20e - thread.cc:325
impalad ! operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0> + 0x5a - bind.hpp:457
impalad ! boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>::operator() - bind_template.hpp:20
impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x19 - thread.hpp:116
impalad ! thread_proxy + 0xd9 - [unknown source file]
libpthread.so.0 ! start_thread + 0xd0 - [unknown source file]
libc.so.6 ! clone + 0x6c - [unknown source file]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)