[jira] [Created] (IMPALA-6816) Statestore spends a lot of time in GetMinSubscriberTopicVersion()

["Tim Armstrong (JIRA)" <ji...@apache.org>]

Tim Armstrong created IMPALA-6816:
-------------------------------------

             Summary: Statestore spends a lot of time in GetMinSubscriberTopicVersion()
                 Key: IMPALA-6816
                 URL: https://issues.apache.org/jira/browse/IMPALA-6816
             Project: IMPALA
          Issue Type: Bug
          Components: Distributed Exec
    Affects Versions: Impala 3.0, Impala 2.12.0
            Reporter: Tim Armstrong
            Assignee: Tim Armstrong


{noformat}
Samples: 13K of event 'cycles', Event count (approx.): 1200870513
  20.23%      statestored  impalad              [.] impala::Statestore::GetMinSubscriberTopicVersion(std::string const&, std::string*)
   7.68%      statestored  [kernel.kallsyms]    [k] find_busiest_group
   3.46%      statestored  impalad              [.] impala::Statestore::Subscriber::LastTopicVersionProcessed(std::string const&) const
   3.26%      statestored  libc-2.12.so         [.] __memcmp_sse4_1
   1.41%      statestored  [kernel.kallsyms]    [k] find_next_bit
   1.40%      statestored  [kernel.kallsyms]    [k] cpumask_next_and
   1.21%      statestored  libpthread-2.12.so   [.] pthread_mutex_lock
   1.04%      statestored  libc-2.12.so         [.] memcpy
   1.01%      statestored  [kernel.kallsyms]    [k] _spin_lock
   0.98%      statestored  impalad              [.] 0x000000000088f903
   0.93%      statestored  impalad              [.] 0x000000000088f8f5
   0.91%      statestored  impalad              [.] 0x000000000088f8ea
   0.85%      statestored  [kernel.kallsyms]    [k] ixgbe_xmit_frame_ring
   0.77%      statestored  impalad              [.] 0x000000000088f8e3
   0.75%      statestored  impalad              [.] 0x000000000088f900
   0.75%      statestored  impalad              [.] impala::Statestore::IsPrioritizedTopic(std::string const&)
   0.73%      statestored  impalad              [.] 0x000000000088f8fa
   0.72%      statestored  impalad              [.] operator new[](unsigned long)
   0.68%      statestored  [kernel.kallsyms]    [k] tcp_recvmsg
   0.67%      statestored  impalad              [.] 0x000000000088f8fd
   0.66%      statestored  impalad              [.] impala::Statestore::Topic::BuildDelta(std::string const&, long, impala::TTopicDelta*)
   0.61%      statestored  [kernel.kallsyms]    [k] thread_return
   0.60%      statestored  impalad              [.] 0x000000000088f8f2
   0.60%      statestored  libstdc++.so.6       [.] std::string::compare(std::string const&) const
   0.59%      statestored  impalad              [.] 0x000000000088f8e6
   0.56%      statestored  impalad              [.] 0x000000000088f8ee
   0.56%      statestored  libcrypto.so.1.0.1e  [.] aesni_encrypt
   0.55%      statestored  impalad              [.] 0x000000000088f8e0
   0.55%      statestored  [kernel.kallsyms]    [k] tcp_transmit_skb
   0.53%      statestored  [kernel.kallsyms]    [k] fget_light
   0.51%      statestored  impalad              [.] std::_Rb_tree<void*, std::pair<void* const, std::shared_ptr<impala::ThriftClientImpl> >, std::_Select1st<std::pair<void* const, std::shared_ptr<impala::ThriftCl
   0.50%      statestored  impalad              [.] apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TBufferedTransport, apache::thrift::transport::TBufferBase>::readAll_virt(unsigned char*
   0.50%      statestored  impalad              [.] impala::Statestore::DoSubscriberUpdate(impala::Statestore::UpdateKind, int, impala::Statestore::ScheduledSubscriberUpdate const&)
   0.49%      statestored  libssl.so.1.0.1e     [.] tls1_enc
   0.48%      statestored  libssl.so.1.0.1e     [.] ssl3_read_bytes
{noformat}

We are spending most of our time computing this for non-catalog topics, where it's not even used.

There are a couple of ways we could fix this that I can think of:
* Avoid including this information for topics where we're not interested in it
* Cache or precompute the value somehow to avoid iterating over all subscribers every time



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)