You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Riza Suminto (Jira)" <ji...@apache.org> on 2023/09/19 18:48:00 UTC
[jira] [Comment Edited] (IMPALA-12018) Consider runtime filters in resource estimates
[ https://issues.apache.org/jira/browse/IMPALA-12018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766904#comment-17766904 ]
Riza Suminto edited comment on IMPALA-12018 at 9/19/23 6:47 PM:
----------------------------------------------------------------
I have an idea that if a scan node receive a runtime filter from a join node, then that join node selectivity can be applied to reduce the cardinality of the scan node. However, this comes with 2 requirement:
# Runtime filter arrived ontime, or it is guaranteed that scan node will need to wait for that runtime filter arrival (ie., join node right above the scan will not start pulling rows before its join build complete).
# The runtime filter itself accurate selectivity.
The second point can be tricky given that Impala's default RUNTIME_FILTER_ERROR_RATE == max_filter_error_rate == 0.75, and join build cardinality itself can be underestimated, leading to undersize bloom filter (IMPALA-12451, IMPALA-12454).
was (Author: rizaon):
I have an idea that if a scan node receive a runtime filter from a join node, then that join node selectivity can be applied to reduce the cardinality of the scan node. However, this comes with 2 requirement:
# Runtime filter arrived ontime, or it is guaranteed that scan node will need to wait for that runtime filter arrival (ie., join node right above the scan will not start pulling rows before its join build complete).
# The runtime filter itself accurate selectivity.
The second point can be tricky given that Impala's default RUNTIME_FILTER_ERROR_RATE == max_filter_error_rate == 0.75, and join build cardinality itself can be underestimated, leading to undersize bloom filter (IMPALA-12451).
> Consider runtime filters in resource estimates
> ----------------------------------------------
>
> Key: IMPALA-12018
> URL: https://issues.apache.org/jira/browse/IMPALA-12018
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Csaba Ringhofer
> Assignee: Riza Suminto
> Priority: Major
>
> Currently Impala creates a plan first and looks for runtime filters bases on the complete plan.
> IMPALA-3573 is about considering runtime filters during join ordering which would be a major change. Meanwhile it could be also useful to consider selective looking runtime filters in resource estimates without changing the plan topology.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org