You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2019/11/01 22:32:59 UTC
[impala] branch master updated (770e72d -> b28d56f)
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 770e72d IMPALA-9105: Catalog debug page top-n table has a URL generation issue
new eb8f415 IMPALA-9073: fix test_executor_concurrency flakiness
new b28d56f IMPALA-9116: KUDU-2989. Work around SASL bug when FQDN is >=64 characters
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/kudu/rpc/server_negotiation.cc | 15 ++++++++++++++-
tests/custom_cluster/test_executor_groups.py | 23 +++++++++++++++--------
2 files changed, 29 insertions(+), 9 deletions(-)
[impala] 02/02: IMPALA-9116: KUDU-2989. Work around SASL bug when
FQDN is >=64 characters
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit b28d56f5b42acef8f95890d647700ef9b8c7ea66
Author: Todd Lipcon <to...@apache.org>
AuthorDate: Thu Oct 31 22:24:10 2019 -0700
IMPALA-9116: KUDU-2989. Work around SASL bug when FQDN is >=64 characters
This adds a workaround for an upstream SASL bug which is triggered when
the FQDN has more than 64 characters. In this case, SASL would truncate
the FQDN and not be able to find the relevant keytab.
The workaround simply uses our own code to determine the FQDN.
Change-Id: I9f05f70915ed20c97efd0ae7295b181a010cf0f6
Change-Id: I4898814f2f7ab87151798336414dde7078d28a4a
Reviewed-on: http://gerrit.cloudera.org:8080/14609
Reviewed-by: Anurag Mantripragada <an...@cloudera.com>
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/14614
Reviewed-by: Todd Lipcon <to...@apache.org>
Reviewed-by: Michael Ho <kw...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
be/src/kudu/rpc/server_negotiation.cc | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/be/src/kudu/rpc/server_negotiation.cc b/be/src/kudu/rpc/server_negotiation.cc
index 612701f..a63d856 100644
--- a/be/src/kudu/rpc/server_negotiation.cc
+++ b/be/src/kudu/rpc/server_negotiation.cc
@@ -380,12 +380,25 @@ Status ServerNegotiation::InitSaslServer() {
unsigned secflags = 0;
sasl_conn_t* sasl_conn = nullptr;
+
+ const char* server_fqdn = helper_.server_fqdn();
+ // If not explicitly set, use the host's FQDN here.
+ // SASL handles this itself if we pass null, but in a buggy way[1] that fails
+ // if the FQDN is >64 characters.
+ //
+ // [1] https://github.com/cyrusimap/cyrus-sasl/issues/583
+ string default_server_fqdn;
+ if (server_fqdn == nullptr) {
+ RETURN_NOT_OK_PREPEND(GetFQDN(&default_server_fqdn), "could not determine own FQDN");
+ server_fqdn = default_server_fqdn.c_str();
+ }
+
RETURN_NOT_OK_PREPEND(WrapSaslCall(nullptr /* no conn */, [&]() {
return sasl_server_new(
// Registered name of the service using SASL. Required.
sasl_proto_name_.c_str(),
// The fully qualified domain name of this server.
- helper_.server_fqdn(),
+ server_fqdn,
// Permits multiple user realms on server. NULL == use default.
nullptr,
// Local and remote IP address strings. We don't use any mechanisms
[impala] 01/02: IMPALA-9073: fix test_executor_concurrency flakiness
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit eb8f415f7c93eaac9e712ea245ebd1a8f6338c01
Author: Tim Armstrong <ta...@cloudera.com>
AuthorDate: Thu Oct 31 15:49:25 2019 -0700
IMPALA-9073: fix test_executor_concurrency flakiness
The test was checking the incorrect invariant - the
slot mechanism only prevents more than than number of
queries running on a backend. More queries can run on
a cluster since the query's backends are freed up before
the query itself finishes.
It was a little tricky picking an appropriate metric
since there is no strong consistency between the
metrics, e.g. decrementing a metric after a backend
finishes may race with admitting the next query.
So I simply used the same metric used by the admission
controller in making decisions, which should be
strongly consistent w.r.t. admission control decissions.
Also remove the concurrency limit on the coordinator,
which seemed inconsistent with the purpose of the
test, because we only want concurrency to be limited
by the executors.
Testing:
Looped the test for a bit.
Change-Id: I910028919f248a3bf5de345e9eade9dbc4353ebd
Reviewed-on: http://gerrit.cloudera.org:8080/14606
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
tests/custom_cluster/test_executor_groups.py | 23 +++++++++++++++--------
1 file changed, 15 insertions(+), 8 deletions(-)
diff --git a/tests/custom_cluster/test_executor_groups.py b/tests/custom_cluster/test_executor_groups.py
index da3beee..1bb070a 100644
--- a/tests/custom_cluster/test_executor_groups.py
+++ b/tests/custom_cluster/test_executor_groups.py
@@ -20,6 +20,7 @@
from tests.common.custom_cluster_test_suite import CustomClusterTestSuite
from tests.util.concurrent_workload import ConcurrentWorkload
+import json
import logging
import pytest
from time import sleep
@@ -261,7 +262,6 @@ class TestExecutorGroups(CustomClusterTestSuite):
client.cancel(q2)
@pytest.mark.execute_serially
- @CustomClusterTestSuite.with_args(impalad_args="-admission_control_slots=3")
def test_executor_concurrency(self):
"""Tests that the command line flag to limit query concurrency on executors works as
expected."""
@@ -282,16 +282,23 @@ class TestExecutorGroups(CustomClusterTestSuite):
for _ in range(RAMP_UP_TIMEOUT_S)), \
"Did not admit enough queries within %s s" % RAMP_UP_TIMEOUT_S
- # Sample the number of running queries for while
- NUM_RUNNING_SAMPLES = 30
- num_running = []
- for _ in xrange(NUM_RUNNING_SAMPLES):
- num_running.append(self._get_num_running_queries())
+ # Sample the number of admitted queries on each backend for while.
+ # Note that the total number of queries in the cluster can higher
+ # than 3 because resources may be released on some backends, allowing
+ # a new query to fit (see IMPALA-9073).
+ NUM_SAMPLES = 30
+ executor_slots_in_use = []
+ for _ in xrange(NUM_SAMPLES):
+ backends_json = json.loads(
+ self.impalad_test_service.read_debug_webpage('backends?json'))
+ for backend in backends_json['backends']:
+ if backend['is_executor']:
+ executor_slots_in_use.append(backend['admission_slots_in_use'])
sleep(1)
# Must reach 3 but not exceed it
- assert max(num_running) == 3, \
- "Unexpected number of running queries: %s" % num_running
+ assert max(executor_slots_in_use) == 3, \
+ "Unexpected number of slots in use: %s" % executor_slots_in_use
finally:
LOG.info("Stopping workload")