You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2020/08/11 16:45:38 UTC
[impala] branch master updated (f95f794 -> ac63e19)
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from f95f794 IMPALA-10017: Implement ds_kll_union() function
new 19f16a0 Fix concurrency for docker-based tests on 140+GB memory machines
new ac63e19 IMPALA-10043: Keep more logs when using EE_TEST_SHARDS
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/run-all-tests.sh | 5 +++++
docker/test-with-docker.py | 1 +
2 files changed, 6 insertions(+)
[impala] 02/02: IMPALA-10043: Keep more logs when using
EE_TEST_SHARDS
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit ac63e19e0d3c797b08dcf80053fc8e3259d8472d
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Wed Aug 5 14:17:54 2020 -0700
IMPALA-10043: Keep more logs when using EE_TEST_SHARDS
IMPALA-9887 introduced the EE_TEST_SHARDS setting, which splits
the end-to-end test into shards and restarts Impala in between.
In order to keep the logs from all the shards, the value for
max_log_files needs to be increased so that none get aged out.
This multiplies IMPALA_MAX_LOG_FILES by the number of shards
using EE_TEST_SHARDS.
Testing:
- Ran a test with EE_TEST_SHARDS=6 and verified that the
logs are preserved.
Change-Id: Ie011b892cd2eb1a528012ec5600e72e44f281a88
Reviewed-on: http://gerrit.cloudera.org:8080/16297
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Laszlo Gaal <la...@cloudera.com>
---
bin/run-all-tests.sh | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/bin/run-all-tests.sh b/bin/run-all-tests.sh
index 5287861..74f65a9 100755
--- a/bin/run-all-tests.sh
+++ b/bin/run-all-tests.sh
@@ -254,6 +254,10 @@ do
# Some test frameworks (e.g. the docker-based tests) use this.
run_ee_tests
else
+ # Increase the maximum number of log files so that the logs from the shards
+ # don't get aged out. Multiply the default number by the number of shards.
+ IMPALA_MAX_LOG_FILES_SAVE="${IMPALA_MAX_LOG_FILES:-10}"
+ export IMPALA_MAX_LOG_FILES="$((${EE_TEST_SHARDS} * ${IMPALA_MAX_LOG_FILES_SAVE}))"
# When the EE tests are sharded, it runs 1/Nth of the tests at a time, restarting
# Impala between the shards. There are two benefits:
# 1. It isolates errors so that if Impala crashes, the next shards will still run
@@ -268,6 +272,7 @@ do
run_ee_tests "--shard_tests=$shard_idx/${EE_TEST_SHARDS}"
start_impala_cluster
done
+ export IMPALA_MAX_LOG_FILES="${IMPALA_MAX_LOG_FILES_SAVE}"
fi
fi
[impala] 01/02: Fix concurrency for docker-based tests on 140+GB
memory machines
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 19f16a0f4889a59f7785bb88d059d2d8c335988d
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Sun Aug 9 19:38:42 2020 -0700
Fix concurrency for docker-based tests on 140+GB memory machines
A prior change increased the suite concurrency for the
docker-based tests on machines with 140+GB of memory.
This new rung should also bump the parallel test
concurrency (i.e. for parallel EE tests). This sets
the parallel test concurrency to 12 for this rung
(which is what we use for the 95GB-140GB rung).
Testing:
- Ran test-with-docker.py on a m5.12xlarge
Change-Id: Ib7299abd585da9ba1a838640dadc0bef9c72a39b
Reviewed-on: http://gerrit.cloudera.org:8080/16326
Reviewed-by: Laszlo Gaal <la...@cloudera.com>
Tested-by: Joe McDonnell <jo...@cloudera.com>
---
docker/test-with-docker.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/docker/test-with-docker.py b/docker/test-with-docker.py
index b348d3b..35f64aa 100755
--- a/docker/test-with-docker.py
+++ b/docker/test-with-docker.py
@@ -253,6 +253,7 @@ def _compute_defaults():
if total_memory_gb >= 140:
suite_concurrency = 6
memlimit_gb = 11
+ parallel_test_concurrency = min(cpus, 12)
elif total_memory_gb >= 95:
suite_concurrency = 4
memlimit_gb = 11