[impala] branch master updated: IMPALA-11659: Avoid statestored sending the host stats to executors

[st...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new 5f7210a8d IMPALA-11659: Avoid statestored sending the host stats to executors
5f7210a8d is described below

commit 5f7210a8df60a7d1a58742b00b3a37c7a27a91de
Author: ttttttz <24...@qq.com>
AuthorDate: Thu Nov 10 10:22:40 2022 +0800

    IMPALA-11659: Avoid statestored sending the host stats to executors
    
    This patch is used to solve the problem that the outbound traffic of
    the statestore is too large due to the statestore broadcast hosts stats
    to all impalads in a large cluster.
    
    Change-Id: I293c63e71c587b0b217f7b13c1931034ce390f62
    Reviewed-on: http://gerrit.cloudera.org:8080/19227
    Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
    Reviewed-by: Quanlong Huang <hu...@gmail.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 be/src/scheduling/admission-controller.cc | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/be/src/scheduling/admission-controller.cc b/be/src/scheduling/admission-controller.cc
index 1e31813ba..1f0c80dfd 100644
--- a/be/src/scheduling/admission-controller.cc
+++ b/be/src/scheduling/admission-controller.cc
@@ -60,6 +60,9 @@ DEFINE_int64_hidden(admission_control_stale_topic_threshold_ms, 5 * 1000,
     "capture most cases where the Impala daemon is disconnected from the statestore "
     "or topic updates are seriously delayed.");
 
+DECLARE_bool(is_coordinator);
+DECLARE_bool(is_executor);
+
 namespace impala {
 
 const int64_t AdmissionController::PoolStats::HISTOGRAM_NUM_OF_BINS = 128;
@@ -660,9 +663,13 @@ Status AdmissionController::Init() {
   auto cb = [this](
       const StatestoreSubscriber::TopicDeltaMap& state,
       vector<TTopicDelta>* topic_updates) { UpdatePoolStats(state, topic_updates); };
+  // The executor only needs to read the entry with the key prefix "POOL:" from the topic.
+  // This can effectively reduce the network load of the statestore.
+  string filter_prefix =
+      FLAGS_is_executor && !FLAGS_is_coordinator ? TOPIC_KEY_POOL_PREFIX : "";
   Status status = subscriber_->AddTopic(Statestore::IMPALA_REQUEST_QUEUE_TOPIC,
       /* is_transient=*/true, /* populate_min_subscriber_topic_version=*/false,
-      /* filter_prefix=*/"", cb);
+      filter_prefix, cb);
   if (!status.ok()) {
     status.AddDetail("AdmissionController failed to register request queue topic");
   }