You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2026/10/23 19:39:25 UTC
[Impala-ASF-CR] PROTOTYPE: Don't evaluate partition column runtime filters per row
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/25202
Change subject: PROTOTYPE: Don't evaluate partition column runtime filters per row
......................................................................
PROTOTYPE: Don't evaluate partition column runtime filters per row
Parquet and ORC scanners evaluate every runtime filter on every row,
including filters that only read partition columns. Such a filter has
the same result for all rows of a partition, and the scan already
applies it to the partition values before each file, split and row
batch. The scanners also check filter effectiveness only every 16
output batches, which selective scans rarely reach in a scan range.
HdfsScanPlanNode now marks the runtime filters of non-Iceberg tables
whose target expression only reads partition key slots, and the
interpreted and codegen'd EvalRuntimeFilters() skip them, so their
row counters in the profile stay 0. Columnar scanners check filter
effectiveness after every scratch batch, and a filter is disabled for
rejecting too few rows only after it was evaluated on at least
--runtime_filter_effectiveness_check_rows rows (new flag, default
16384). A value <= 0 restores the check every 16 row batches.
Testing:
- TPC-DS SF100, 5 executors, MT_DOP=0: total -4.2% (-1.8% without
q72 and q78), geomean 0.94.
- No new tests. Existing tests were not run.
Change-Id: I66de7020d121acb360f6d1924671d1ff4e70c438
Assisted-by: Claude Opus 5.5 (Claude Code)
---
M be/src/exec/hdfs-columnar-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
5 files changed, 119 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/25202/1
--
To view, visit http://gerrit.cloudera.org:8080/25202
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I66de7020d121acb360f6d1924671d1ff4e70c438
Gerrit-Change-Number: 25202
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] PROTOTYPE: Don't evaluate partition column runtime filters per row
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25202 )
Change subject: PROTOTYPE: Don't evaluate partition column runtime filters per row
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24662/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25202
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I66de7020d121acb360f6d1924671d1ff4e70c438
Gerrit-Change-Number: 25202
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 23 Oct 2026 20:06:43 +0000
Gerrit-HasComments: No