You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/10/20 23:55:00 UTC
[jira] [Commented] (IMPALA-14510) Runtime filters with low effectiveness should be disabled more aggressively
[ https://issues.apache.org/jira/browse/IMPALA-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129375#comment-18129375 ]
ASF subversion and git services commented on IMPALA-14510:
----------------------------------------------------------
Commit 867c237b5dc62cbfa84be0bdcadec517c1b7bfd8 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=867c237b5d ]
IMPALA-14510: Disable ineffective runtime filters more aggressively
HdfsScanner::GetNext() does a check for runtime filter effectiveness
every 16 row batches. When runtime filters are evaluated at the row
level, a highly selective runtime filter may eliminate so many rows
that GetNext() is rarely returning a row batch. When an ineffective
runtime filter is paired with this highly selective filter, the
ineffective filter may never get disabled and could be evaluated on
millions of rows with no benefit. For example, on a run of TPC-DS Q63,
we see this in the profile:
- Rows processed: 5.22M (5216355)
- Rows rejected: 0 (0)
- Rows total: 5.96M (5955191)
This filter isn't getting disabled quickly, because it is paired
with a highly effective filter on the same scan node:
- Rows processed: 5.96M (5955191)
- Rows rejected: 5.44M (5442260)
- Rows total: 5.96M (5955191)
This adds an additional check for runtime filter effectiveness that
runs every 32k unfiltered rows. This means that ineffective filters
get disabled even if they are paired with a highly selective filter.
This disables some ineffective filters on both TPC-H and TPC-DS
without a noticeable overhead. Runtime filters are disabled
independently for each file, so there may still be some room for
further improvement by remembering bad filters across scan range
boundaries.
Testing:
- Ran TPC-DS with multiple settings for the threshold.
32k outperformed 64k, but there were no further improvements
at 16k.
Change-Id: Iab48fe610b7784ea34c0df729e4e5207712efedd
Reviewed-on: http://gerrit.cloudera.org:8080/23595
Reviewed-by: Arnab Karmakar <ar...@gmail.com>
Reviewed-by: Jason Fehr <jf...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Runtime filters with low effectiveness should be disabled more aggressively
> ---------------------------------------------------------------------------
>
> Key: IMPALA-14510
> URL: https://issues.apache.org/jira/browse/IMPALA-14510
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
> Attachments: disable_ineffective_filters_tpcds.txt
>
>
> There is an existing check for a runtime filter's effectiveness in HdfsScanner::CheckFiltersEffectiveness() which can disable runtime filters that are always true or don't meet the min_filter_reject_ratio (default value 0.1). This check runs every BATCHES_PER_FILTER_SELECTIVITY_CHECK (16) batches:
> {noformat}
> // Always add batch to the queue because it may contain data referenced by previously
> // appended batches.
> scan_node->AddMaterializedRowBatch(move(batch));
> RETURN_IF_ERROR(status);
> ++row_batches_produced_;
> if ((row_batches_produced_ & (BATCHES_PER_FILTER_SELECTIVITY_CHECK - 1)) == 0) {
> CheckFiltersEffectiveness();
> }{noformat}
> If there are multiple runtime filters and one of them is very selective, it may not be returning row batches very often. This means that an ineffective filter can be evaluated on many rows before being disabled (or may never be disabled). For example from TPC-DS Q13:
> {noformat}
> - Rows processed: 47.46M (47458959)
> - Rows rejected: 0 (0)
> - Rows total: 47.46M (47458959)
> {noformat}
> We should try to disable ineffective filters more aggressively in this circumstance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org