You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/03/03 00:39:00 UTC
[jira] [Commented] (IMPALA-10492) Lower default MAX_CNF_EXPRS query
option
[ https://issues.apache.org/jira/browse/IMPALA-10492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294075#comment-17294075 ]
ASF subversion and git services commented on IMPALA-10492:
----------------------------------------------------------
Commit 801efdcf097e82c79a077141c753d6d4d44d83e8 in impala's branch refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=801efdc ]
IMPALA-10492: Lower default MAX_CNF_EXPRS query option
MAX_CNF_EXPRS was set to unlimited by default. The CNF rewrite can lead
to significant frontend memory usage and eventually OutOfMemory for a
complex query that contain many predicates. We need to lower the default
value to avoid this memory problem while maintaining performance for our
TPC-DS and TPC-H workloads.
We investigate the maximum number of CNF expressions in TPC-DS and TPC-H
by printing out the final value of 'numCnfExprs_' from
ConvertToCNFRule.java to the query profile. We found 5 queries that
applies CNF rewrite rules as follow:
| Query | numCnfExprs_ |
|-----------+--------------|
| TPCDS-Q13 | 168 |
| TPCDS-Q85 | 100 |
| TPCDS-Q48 | 34 |
| TPCH-Q19 | 124 |
| TPCH-Q7 | 3 |
This patch lower the default value from unlimited to 200 based on the
result above.
Testing:
- Manually verify that MAX_CNF_EXPRS 200 is enough for our TPC-DS and
TPC-H worloads.
- Pass core tests.
Change-Id: I7ca3d0e094ac01c24a046c25d6a1b56bf134faa8
Reviewed-on: http://gerrit.cloudera.org:8080/17132
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Lower default MAX_CNF_EXPRS query option
> ----------------------------------------
>
> Key: IMPALA-10492
> URL: https://issues.apache.org/jira/browse/IMPALA-10492
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Affects Versions: Impala 4.0
> Reporter: Tamas Mate
> Assignee: Riza Suminto
> Priority: Minor
> Labels: ramp-up
>
> Currently MAX_CNF_EXPRS is set to unlimited by default, with a complex querie that container many predicates the CNF rewrite can lead to significant frontend memory usage and eventually OutOfMemory. A potential default value could be in the range of 1000, but it should work well with the current TCP-DS queries.
> cc.: [~amansinha], [~drorke]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org