You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2022/09/09 05:21:39 UTC
[impala] branch branch-4.1.1 updated (5cae46a4b -> 9cd363e06)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch branch-4.1.1
in repository https://gitbox.apache.org/repos/asf/impala.git
from 5cae46a4b IMPALA-11281: Load table metadata for ResetMetadataStmt
new 96fcc7286 IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
new 2aeb6013f IMPALA-7864: (Addendum) Deflake test_replan_limit by postponing catalog fetches
new 9cd363e06 Pin to asf-impala-4.1 branch of native-toolchain
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/catalog-op-executor.cc | 5 +++++
bin/bootstrap_system.sh | 7 ++++---
tests/custom_cluster/test_local_catalog.py | 12 ++++++++----
3 files changed, 17 insertions(+), 7 deletions(-)
[impala] 03/03: Pin to asf-impala-4.1 branch of native-toolchain
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch branch-4.1.1
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 9cd363e060547db108f7536167dfc19f17eb5ccb
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Fri Aug 26 09:16:10 2022 +0800
Pin to asf-impala-4.1 branch of native-toolchain
Change-Id: Ic65d9d006b31632868786373fbaa19ef023e007a
Reviewed-on: http://gerrit.cloudera.org:8080/18952
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Quanlong Huang <hu...@gmail.com>
---
bin/bootstrap_system.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index 17fc70efd..958ff070a 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -494,7 +494,7 @@ if [[ $ARCH_NAME == 'aarch64' ]]; then
echo -e "\n$SET_TOOLCHAIN_HOME" >> "${IMPALA_HOME}/bin/impala-config-local.sh"
eval "$SET_TOOLCHAIN_HOME"
if ! [[ -d "$NATIVE_TOOLCHAIN_HOME" ]]; then
- time -p git clone https://github.com/cloudera/native-toolchain/ \
+ time -p git clone -b asf-impala-4.1 https://github.com/cloudera/native-toolchain/ \
"$NATIVE_TOOLCHAIN_HOME"
fi
cd "$NATIVE_TOOLCHAIN_HOME"
[impala] 01/03: IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch branch-4.1.1
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 96fcc7286b59101f4aa2b10f5fbe66fe0c9065cb
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Fri Jul 22 12:53:08 2022 -0700
IMPALA-11492: Add langpacks-en (centos) and language-pack-en (Ubuntu)
Machines that don't have en_US.UTF-8 installed see
issues when running ExprTest.Utf8MaskTest.
This currently impacts the Docker-based tests.
This installs the appropriate language packs
to have en_US.UTF-8 installed.
Testing:
- Ran docker-based tests and verified that
ExprTest.Utf8MaskTest passes.
Change-Id: I1b8696190e4713bda787e773d48943b5dfc6335e
Reviewed-on: http://gerrit.cloudera.org:8080/18875
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Reviewed-on: http://gerrit.cloudera.org:8080/18950
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Quanlong Huang <hu...@gmail.com>
---
bin/bootstrap_system.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index ec7919f77..17fc70efd 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -219,7 +219,8 @@ ubuntu apt-get --yes install ccache curl gawk g++ gcc libffi-dev \
libkrb5-dev krb5-admin-server krb5-kdc krb5-user libsasl2-dev \
libsasl2-modules libsasl2-modules-gssapi-mit libssl-dev make ninja-build \
python-dev python-setuptools postgresql ssh wget vim-common psmisc \
- lsof openjdk-8-jdk openjdk-8-source openjdk-8-dbg apt-utils git ant
+ lsof openjdk-8-jdk openjdk-8-source openjdk-8-dbg apt-utils git ant \
+ language-pack-en
# Required by Kudu in the minicluster
ubuntu20 apt-get --yes install libtinfo5
ARCH_NAME=$(uname -p)
@@ -266,7 +267,7 @@ redhat sudo yum install -y curl gawk gcc gcc-c++ git krb5-devel krb5-server \
postgresql postgresql-server \
wget vim-common nscd cmake fuse-devel zlib-devel \
psmisc lsof openssh-server redhat-lsb java-1.8.0-openjdk-devel \
- java-1.8.0-openjdk-src
+ java-1.8.0-openjdk-src langpacks-en
# Enable the Powertools repo for snappy-devel on RedHat 8
redhat8 sudo yum install -y dnf-plugins-core
[impala] 02/03: IMPALA-7864: (Addendum) Deflake test_replan_limit by postponing catalog fetches
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch branch-4.1.1
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 2aeb6013fa44e53031d82b7e7ca59d771037d60a
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Wed May 18 16:51:37 2022 +0800
IMPALA-7864: (Addendum) Deflake test_replan_limit by postponing catalog fetches
TestLocalCatalogRetries.test_replan_limit runs REFRESH and SELECT
queries concurrently on a table, and expects one of the query hits
inconsistent metadata.
This patch increases the chance of inconsistent metadata by injecting
a latency (500ms) before each catalog fetch. So it's more likely that a
request is fetching stale metadata. Also bump up the timeout of
thread.join() so we can try out all the attempts.
Test
- Run test_replan_limit 1000 times without any error.
- Run all tests of TestLocalCatalogRetries 100 times without any error.
Change-Id: Ia5bdca7402039f1f24b7bf19595c2541fa32d0ad
Reviewed-on: http://gerrit.cloudera.org:8080/18537
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-on: http://gerrit.cloudera.org:8080/18951
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Quanlong Huang <hu...@gmail.com>
---
be/src/exec/catalog-op-executor.cc | 5 +++++
tests/custom_cluster/test_local_catalog.py | 12 ++++++++----
2 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/be/src/exec/catalog-op-executor.cc b/be/src/exec/catalog-op-executor.cc
index 646e6aa52..c6c245428 100644
--- a/be/src/exec/catalog-op-executor.cc
+++ b/be/src/exec/catalog-op-executor.cc
@@ -55,6 +55,8 @@ DECLARE_int32(catalog_client_connection_num_retries);
DECLARE_int32(catalog_client_rpc_timeout_ms);
DECLARE_int32(catalog_client_rpc_retry_interval_ms);
+DEFINE_int32_hidden(inject_latency_before_catalog_fetch_ms, 0,
+ "Latency (ms) to be injected before fetching catalog data from the catalogd");
DEFINE_int32_hidden(inject_latency_after_catalog_fetch_ms, 0,
"Latency (ms) to be injected after fetching catalog data from the catalogd");
@@ -366,6 +368,9 @@ Status CatalogOpExecutor::GetPartialCatalogObject(
DCHECK(FLAGS_use_local_catalog || TestInfo::is_test());
const TNetworkAddress& address =
MakeNetworkAddress(FLAGS_catalog_service_host, FLAGS_catalog_service_port);
+ if (FLAGS_inject_latency_before_catalog_fetch_ms > 0) {
+ SleepForMs(FLAGS_inject_latency_before_catalog_fetch_ms);
+ }
int attempt = 0; // Used for debug action only.
CatalogServiceConnection::RpcStatus rpc_status =
CatalogServiceConnection::DoRpcWithRetry(env_->catalogd_client_cache(), address,
diff --git a/tests/custom_cluster/test_local_catalog.py b/tests/custom_cluster/test_local_catalog.py
index 63b0cbb91..6e74a4de0 100644
--- a/tests/custom_cluster/test_local_catalog.py
+++ b/tests/custom_cluster/test_local_catalog.py
@@ -273,8 +273,9 @@ class TestLocalCatalogRetries(CustomClusterTestSuite):
q = random.choice(queries)
attempt += 1
try:
+ print 'Attempt', attempt, 'client', str(client)
ret = self.execute_query_unchecked(client, q)
- except Exception, e:
+ except Exception as e:
if 'InconsistentMetadataFetchException' in str(e):
with inconsistent_seen_lock:
inconsistent_seen[0] += 1
@@ -287,7 +288,8 @@ class TestLocalCatalogRetries(CustomClusterTestSuite):
t.start()
for t in threads:
# When there are failures, they're observed quickly.
- t.join(30)
+ # 600s is enough for 200 attempts.
+ t.join(600)
assert failed_queries.empty(),\
"Failed query count non zero: %s" % list(failed_queries.queue)
@@ -318,7 +320,8 @@ class TestLocalCatalogRetries(CustomClusterTestSuite):
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args(
- impalad_args="--use_local_catalog=true --local_catalog_max_fetch_retries=0",
+ impalad_args="--use_local_catalog=true --local_catalog_max_fetch_retries=0"
+ " --inject_latency_before_catalog_fetch_ms=500",
catalogd_args="--catalog_topic_mode=minimal")
def test_replan_limit(self):
"""
@@ -326,7 +329,8 @@ class TestLocalCatalogRetries(CustomClusterTestSuite):
an inconsistent metadata exception when running concurrent reads/writes
is seen. With the max retries set to 0, no retries are expected and with
the concurrent read/write workload, an inconsistent metadata exception is
- expected.
+ expected. Setting inject_latency_before_catalog_fetch_ms to increases the
+ possibility of a stale request which throws the expected exception.
"""
queries = [
'refresh functional.alltypes',