[impala] 04/08: IMPALA-10519: Allow setting of num_reactors for KuduClient

[st...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit c5e8c973221be640063472226a42ee6776dd9f2a
Author: Thomas Tauber-Marshall <tm...@cloudera.com>
AuthorDate: Thu Feb 18 13:04:02 2021 -0800

    IMPALA-10519: Allow setting of num_reactors for KuduClient
    
    The KuduClient allows setting a number of reactor threads, which are
    used for sending rpcs to Kudu such as for scans. The defaults is 4
    threads, which may be too few for many use cases, since there is a
    single KuduClient per impalad.
    
    This patch adds a flag --kudu_client_num_reactor_threads which allows
    users to set the number of threads. The default for the flag is 4,
    which reflects the current default within Kudu, to avoid possible
    regressions.
    
    Some follow up work could be to do performance experiments and set
    the default empirically.
    
    Change-Id: If2ccb9659b9223c9a5de2416b946e6313a3239ff
    Reviewed-on: http://gerrit.cloudera.org:8080/17086
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 be/src/exec/kudu-util.cc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/be/src/exec/kudu-util.cc b/be/src/exec/kudu-util.cc
index 7d9bb34..cb40f69 100644
--- a/be/src/exec/kudu-util.cc
+++ b/be/src/exec/kudu-util.cc
@@ -44,6 +44,10 @@ using kudu::client::KuduColumnTypeAttributes;
 using kudu::client::KuduValue;
 using DataType = kudu::client::KuduColumnSchema::DataType;
 
+// TODO: choose the default empirically
+DEFINE_int32(kudu_client_num_reactor_threads, 4,
+    "Number of threads the Kudu client can use to send rpcs to Kudu. Must be > 0.");
+
 DECLARE_bool(disable_kudu);
 DECLARE_int32(kudu_client_rpc_timeout_ms);
 DECLARE_int32(kudu_client_connection_negotiation_timeout_ms);
@@ -83,6 +87,12 @@ Status CreateKuduClient(const vector<string>& master_addrs,
   }
   b.connection_negotiation_timeout(kudu::MonoDelta::FromMilliseconds(
       FLAGS_kudu_client_connection_negotiation_timeout_ms));
+  if (FLAGS_kudu_client_num_reactor_threads > 0) {
+    b.num_reactors(FLAGS_kudu_client_num_reactor_threads);
+  } else {
+    LOG(WARNING) << "Ignoring value of --kudu_client_num_reactor_threads: "
+                 << FLAGS_kudu_client_num_reactor_threads;
+  }
   KUDU_RETURN_IF_ERROR(b.Build(client), "Unable to create Kudu client");
   return Status::OK();
 }