You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2026/09/25 14:58:38 UTC
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24947
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
IcebergScanPlanner collected the residual expressions of the
FileScanTasks into a TreeSet ordered by
ExpressionUtil.toSanitizedString(). The sanitized string replaces
literal values with placeholders, e.g. both 'id != 12345' and
'id != 54321' become 'id != (5-digit-int)', so the set only kept one
of two different residuals. With ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING
(on by default) the conjunct of the lost residual was then skipped
from the scan node, and the query returned rows that should have been
filtered out. This happens when non-identity partition transforms
(e.g. truncate or day) leave different residuals for different files.
This patch keys the residuals by Expression.toString(), which prints
the literals in full. It uses a TreeMap so the order of the retained
conjuncts stays deterministic. Consecutive files often have the same
residual object (e.g. every file of an unpartitioned table), so the
string conversion is skipped for them. This also removes the cost of
sanitizing the residual of every file during planning. A predicate
that is part of multiple residuals is only retained once.
Testing:
- Added EE tests with truncate and day partition transforms. Both
returned wrong results without the fix.
- Ran PlannerTest and test_iceberg.py.
Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Assisted-by: Claude Opus 5.5 (1M context) <no...@anthropic.com>
---
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-predicate-subsetting.test
M tests/query_test/test_iceberg.py
3 files changed, 56 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/24947/1
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Sep 2026 18:11:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 1: Code-Review+1
Thanks for the fix!
I would have chosen different literals for the tests that are easier to distinguish between visually, but otherwise LGTM
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 12:35:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
IcebergScanPlanner collected the residual expressions of the
FileScanTasks into a TreeSet ordered by
ExpressionUtil.toSanitizedString(). The sanitized string replaces
literal values with placeholders, e.g. both 'id != 11' and
'id != 21' become 'id != (2-digit-int)', so the set only kept one of
two different residuals. With ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING
(on by default) the conjunct of the lost residual was then skipped
from the scan node, and the query returned rows that should have been
filtered out. This happens when non-identity partition transforms
(e.g. truncate or day) leave different residuals for different files.
This patch keys the residuals by Expression.toString(), which prints
the literals in full. It uses a TreeMap so the order of the retained
conjuncts stays deterministic. Consecutive files often have the same
residual object (e.g. every file of an unpartitioned table), so the
string conversion is skipped for them. This also removes the cost of
sanitizing the residual of every file during planning. A predicate
that is part of multiple residuals is only retained once.
Testing:
- Added EE tests with truncate and day partition transforms. Both
returned wrong results without the fix.
- Ran PlannerTest and test_iceberg.py.
Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Assisted-by: Claude Opus 5.5 (1M context) <no...@anthropic.com>
Reviewed-on: http://gerrit.cloudera.org:8080/24947
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-predicate-subsetting.test
M tests/query_test/test_iceberg.py
3 files changed, 56 insertions(+), 9 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24232/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 16:30:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24186/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Sep 2026 15:19:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Sep 2026 13:49:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14170/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Sep 2026 13:49:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 2:
Thanks for the comment, updated the tests based on our discusson
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 16:07:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
Patch Set 2: Code-Review+2
Thanks, LGTM
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Sep 2026 10:39:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Peter Rozsa, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24947
to look at the new patch set (#2).
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
......................................................................
IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting
IcebergScanPlanner collected the residual expressions of the
FileScanTasks into a TreeSet ordered by
ExpressionUtil.toSanitizedString(). The sanitized string replaces
literal values with placeholders, e.g. both 'id != 11' and
'id != 21' become 'id != (2-digit-int)', so the set only kept one of
two different residuals. With ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING
(on by default) the conjunct of the lost residual was then skipped
from the scan node, and the query returned rows that should have been
filtered out. This happens when non-identity partition transforms
(e.g. truncate or day) leave different residuals for different files.
This patch keys the residuals by Expression.toString(), which prints
the literals in full. It uses a TreeMap so the order of the retained
conjuncts stays deterministic. Consecutive files often have the same
residual object (e.g. every file of an unpartitioned table), so the
string conversion is skipped for them. This also removes the cost of
sanitizing the residual of every file during planning. A predicate
that is part of multiple residuals is only retained once.
Testing:
- Added EE tests with truncate and day partition transforms. Both
returned wrong results without the fix.
- Ran PlannerTest and test_iceberg.py.
Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Assisted-by: Claude Opus 5.5 (1M context) <no...@anthropic.com>
---
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-predicate-subsetting.test
M tests/query_test/test_iceberg.py
3 files changed, 56 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/24947/2
--
To view, visit http://gerrit.cloudera.org:8080/24947
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013
Gerrit-Change-Number: 24947
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>