You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2021/06/15 01:54:13 UTC
[impala] branch master updated: IMPALA-9355:
TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory
limit
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 4290c5e IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
4290c5e is described below
commit 4290c5e2971b056f5249716655ef2f704465f383
Author: Qifan Chen <qc...@cloudera.com>
AuthorDate: Fri Jun 11 14:15:36 2021 -0400
IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
This patch restores the previous testing enviroment in which exchange
node 5 in the test query shown below reliablly reports memory
exceeding limit error.
set mem_limit=171520k;
set num_scanner_threads=1;
select *
from tpch_parquet.lineitem l1
join tpch_parquet.lineitem l2 on l1.l_orderkey = l2.l_orderkey and
l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey
and l1.l_linenumber = l2.l_linenumber
order by l1.l_orderkey desc, l1.l_partkey, l1.l_suppkey, l1.l_linenumber
limit 5
In the patch, the memory limit for the entire query is lowered to 164Mb,
less than 171520k, the minimum for the query to be accepted after
the result spooling is enabled by default. This is achieved by
disabling result spooling by setting spool_query_results to false.
Testing:
1). Ran exchange-mem-scaling.test
Change-Id: Ia4ad4508028645b67de419cfdfa2327d2847cfc4
Reviewed-on: http://gerrit.cloudera.org:8080/17586
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../functional-query/queries/QueryTest/exchange-mem-scaling.test | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test b/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
index d80cd8f..10c7a95 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
@@ -4,7 +4,13 @@
# it hits the memory limit in the exchange node when allocating receiver-side
# buffers. It's also possible but less likely that this will hit a memory limit
# in the scan nodes.
-set mem_limit=171520k;
+# IMPALA-9355: An exchange node does not set a memory limit for buffer pool. To
+# hit the memory limit exceeding error reliably, the memory limit for the entire query
+# has to be set to a value less than 171520k (167.5Mb) which is the minimum for the query
+# to be accepted after the result spooling is enabled by default. Disable result spooling
+# here by setting spool_query_results to false, and set the query memory limit to 164Mb.
+set mem_limit=164m;
+set spool_query_results=false;
set num_scanner_threads=1;
select *
from tpch_parquet.lineitem l1