You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2021/06/15 01:54:13 UTC

[impala] branch master updated: IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit

This is an automated email from the ASF dual-hosted git repository.

stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new 4290c5e  IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
4290c5e is described below

commit 4290c5e2971b056f5249716655ef2f704465f383
Author: Qifan Chen <qc...@cloudera.com>
AuthorDate: Fri Jun 11 14:15:36 2021 -0400

    IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
    
    This patch restores the previous testing enviroment in which exchange
    node 5 in the test query shown below reliablly reports memory
    exceeding limit error.
    
      set mem_limit=171520k;
      set num_scanner_threads=1;
      select *
      from tpch_parquet.lineitem l1
        join tpch_parquet.lineitem l2 on l1.l_orderkey = l2.l_orderkey and
            l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey
            and l1.l_linenumber = l2.l_linenumber
      order by l1.l_orderkey desc, l1.l_partkey, l1.l_suppkey, l1.l_linenumber
      limit 5
    
    In the patch, the memory limit for the entire query is lowered to 164Mb,
    less than 171520k, the minimum for the query to be accepted after
    the result spooling is enabled by default. This is achieved by
    disabling result spooling by setting spool_query_results to false.
    
    Testing:
      1). Ran exchange-mem-scaling.test
    
    Change-Id: Ia4ad4508028645b67de419cfdfa2327d2847cfc4
    Reviewed-on: http://gerrit.cloudera.org:8080/17586
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 .../functional-query/queries/QueryTest/exchange-mem-scaling.test  | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test b/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
index d80cd8f..10c7a95 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test
@@ -4,7 +4,13 @@
 # it hits the memory limit in the exchange node when allocating receiver-side
 # buffers. It's also possible but less likely that this will hit a memory limit
 # in the scan nodes.
-set mem_limit=171520k;
+# IMPALA-9355: An exchange node does not set a memory limit for buffer pool. To
+# hit the memory limit exceeding error reliably, the memory limit for the entire query
+# has to be set to a value less than 171520k (167.5Mb) which is the minimum for the query
+# to be accepted after the result spooling is enabled by default. Disable result spooling
+# here by setting spool_query_results to false, and set the query memory limit to 164Mb.
+set mem_limit=164m;
+set spool_query_results=false;
 set num_scanner_threads=1;
 select *
 from tpch_parquet.lineitem l1