You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by ta...@apache.org on 2018/04/19 23:51:20 UTC
[1/4] impala git commit: IMPALA-6917: Limit impalad mem-limit to 12GB.
Repository: impala
Updated Branches:
refs/heads/master 5bbcfaf22 -> b68e06997
IMPALA-6917: Limit impalad mem-limit to 12GB.
This changes the memlimit choosing code in start-impala-cluster to have
a maximum of 12GB. In 68GB machines, this has the effect of reducing the
memlimit from ~15.8GB to 12GB. On machines with less than 51.4 GB of
RAM, this has no effect.
I'm tinkering with this threshold because ASAN builds sometimes
die at the hands of the OOM killer on m2.4xlarge machines (68GB).
My theory for why it's only sometimes is that during the parallel
tests, our memory usage could vary widely depending on which tests
and queries execute.
End-users don't see this code; this is only used by our minicluster
tests to test Impala.
I have run the ASAN build with this change successfully, though I've
found this particular OOM seems to come and go.
Change-Id: I8024414c5c23bb42cce912d8f34cd0b787e0e39a
Reviewed-on: http://gerrit.cloudera.org:8080/10051
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/48643a57
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/48643a57
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/48643a57
Branch: refs/heads/master
Commit: 48643a572b165f0d87c33e6f3a5f88d0b0214d2a
Parents: 5bbcfaf
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Sat Apr 7 20:43:03 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Thu Apr 19 20:31:34 2018 +0000
----------------------------------------------------------------------
bin/start-impala-cluster.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/48643a57/bin/start-impala-cluster.py
----------------------------------------------------------------------
diff --git a/bin/start-impala-cluster.py b/bin/start-impala-cluster.py
index bfe71d5..e870852 100755
--- a/bin/start-impala-cluster.py
+++ b/bin/start-impala-cluster.py
@@ -216,15 +216,21 @@ def start_impalad_instances(cluster_size, num_coordinators, use_exclusive_coordi
# No impalad instances should be started.
return
+ # Set mem_limit of each impalad to the smaller of 12GB or
+ # 1/cluster_size (typically 1/3) of 70% of system memory.
+ #
# The default memory limit for an impalad is 80% of the total system memory. On a
# mini-cluster with 3 impalads that means 240%. Since having an impalad be OOM killed
# is very annoying, the mem limit will be reduced. This can be overridden using the
# --impalad_args flag. virtual_memory().total returns the total physical memory.
# The exact ratio to use is somewhat arbitrary. Peak memory usage during
# tests depends on the concurrency of parallel tests as well as their ordering.
- # At a ratio of 0.8, on 8-core, 68GB machines, ASAN builds can trigger the OOM
- # killer, so this ratio is currently set to 0.7.
+ # On the other hand, to avoid using too much memory, we limit the
+ # memory choice here to max out at 12GB. This should be sufficient for tests.
+ #
+ # Beware that ASAN builds use more memory than regular builds.
mem_limit = int(0.7 * psutil.virtual_memory().total / cluster_size)
+ mem_limit = min(12 * 1024 * 1024 * 1024, mem_limit)
delay_list = []
if options.catalog_init_delays != "":
[4/4] impala git commit: IMPALA-6880: disable flaky bloom filter test
Posted by ta...@apache.org.
IMPALA-6880: disable flaky bloom filter test
This test is made flaky by IMPALA-6338. While that is being worked on,
temporarily disable this test.
Change-Id: I595645b0f2875614294adc7abb4572aec1be8ad5
Reviewed-on: http://gerrit.cloudera.org:8080/10122
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/b68e0699
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/b68e0699
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/b68e0699
Branch: refs/heads/master
Commit: b68e06997c1f49f6b723d78e217efddec4f56f3a
Parents: a12d19c
Author: Thomas Tauber-Marshall <tm...@cloudera.com>
Authored: Thu Apr 19 18:47:08 2018 +0000
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Thu Apr 19 23:43:19 2018 +0000
----------------------------------------------------------------------
.../functional-query/queries/QueryTest/bloom_filters.test | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/b68e0699/testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test
----------------------------------------------------------------------
diff --git a/testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test b/testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test
index 4908e20..b8d896f 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test
@@ -15,7 +15,8 @@ select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a LIMIT 1) a
---- RESULTS
0
---- RUNTIME_PROFILE
-row_regex: .*0 of 1 Runtime Filter Published, 1 Disabled.*
+# TODO: reenable once IMPALA-6338 is fixed
+#row_regex: .*0 of 1 Runtime Filter Published, 1 Disabled.*
====
[3/4] impala git commit: IMPALA-6884: Skip
test_misaligned_orc_stripes on invalid platforms
Posted by ta...@apache.org.
IMPALA-6884: Skip test_misaligned_orc_stripes on invalid platforms
test_misaligned_orc_stripes relies on generating files
with a specific number of HDFS blocks. This is not
possible on S3, ADLS, or Isilon, so the test needs to
be skipped on those platforms. Additionally, it relies
on running multiple impalads, so it needs to be skipped
on local filesystem tests.
Change-Id: I8395b4eb06847a35c630b3ef43e289960d78382a
Reviewed-on: http://gerrit.cloudera.org:8080/10117
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/a12d19c7
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/a12d19c7
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/a12d19c7
Branch: refs/heads/master
Commit: a12d19c794e0d8a9bd3ae62be3718c1b803c33db
Parents: abcc135
Author: Joe McDonnell <jo...@cloudera.com>
Authored: Thu Apr 19 10:30:46 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Thu Apr 19 21:58:48 2018 +0000
----------------------------------------------------------------------
tests/query_test/test_scanners.py | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/a12d19c7/tests/query_test/test_scanners.py
----------------------------------------------------------------------
diff --git a/tests/query_test/test_scanners.py b/tests/query_test/test_scanners.py
index bae52a6..89da580 100644
--- a/tests/query_test/test_scanners.py
+++ b/tests/query_test/test_scanners.py
@@ -912,6 +912,10 @@ class TestOrc(ImpalaTestSuite):
cls.ImpalaTestMatrix.add_constraint(
lambda v: v.get_value('table_format').file_format == 'orc')
+ @SkipIfS3.hdfs_block_size
+ @SkipIfADLS.hdfs_block_size
+ @SkipIfIsilon.hdfs_block_size
+ @SkipIfLocal.multiple_impalad
def test_misaligned_orc_stripes(self, vector, unique_database):
self._build_lineitem_table_helper(unique_database, 'lineitem_threeblocks',
'lineitem_threeblocks.orc')
[2/4] impala git commit: [DOCS] A typo fix introduced in impala-6748
Posted by ta...@apache.org.
[DOCS] A typo fix introduced in impala-6748
Change-Id: Idb89c97d2cc470960b88f14f038252d1d46778d0
Reviewed-on: http://gerrit.cloudera.org:8080/10126
Reviewed-by: Alex Rodoni <ar...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/abcc1356
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/abcc1356
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/abcc1356
Branch: refs/heads/master
Commit: abcc1356ea38e63242d89fd20c269c6a684dc7a1
Parents: 48643a5
Author: Alex Rodoni <ar...@cloudera.com>
Authored: Thu Apr 19 14:04:57 2018 -0700
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Thu Apr 19 21:24:56 2018 +0000
----------------------------------------------------------------------
docs/shared/impala_common.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/abcc1356/docs/shared/impala_common.xml
----------------------------------------------------------------------
diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml
index 75fceb0..6a2315a 100644
--- a/docs/shared/impala_common.xml
+++ b/docs/shared/impala_common.xml
@@ -1410,7 +1410,7 @@ drop database temp;
</p>
<p>Leading zeroes are not required in the numbers representing the date
component, such as month and date, or the time component, such as
- month, date, hour, minute, second. For example, Impala accepts both
+ hour, minute, and second. For example, Impala accepts both
<codeph>"2018-1-1 01:02:03"</codeph> and
<codeph>"2018-01-01 1:2:3"</codeph> as valid.</p>