You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2022/11/23 18:43:59 UTC

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19274


Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................

IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

If an Iceberg V2 table is partitioned, and contains delete files,
then in a query that involves runtime filters on the partition columns
return empty result set.

E.g.:

  select count(*)
  from store_sales, date_dim
  where d_date_sk = ss_sold_date_sk and d_moy=2 and d_year=1998;

In the above query store_sales is partitioned by ss_sold_date_sk which
will be filtered by runtime filters created by the JOIN. If store_sales
has delete files then the above query returns empty result set.

The problem is that we are invoking PartitionPassesFilters() on these
Iceberg tables. It is usually a no-op for Iceberg tables, as the
template tuple is NULL. But when we have virtual columns a template
tuple has been created in HdfsScanPlanNode::InitTemplateTuple. For
Iceberg tables this tempalte tuple is incomplete, i.e. it doesn't
have the partition values set. This means the filters evaluate to
false and the files are getting filtered out, hence the query
produces an empty result set.

With this patch we don't invoke PartitionPassesFilters() on Iceberg
tables, only the Iceberg-specific IcebergPartitionPassesFilters()
gets invoked. Also added DCHECKs to ensure this.

Testing:
 * e2e tests added

Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
5 files changed, 80 insertions(+), 18 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/19274/1
-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Nov 2022 13:11:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8832/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 23 Nov 2022 20:33:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.
Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 2: Code-Review+1

Thanks for fixing this, it made the control flow nicer too, LGTM


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Nov 2022 11:32:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 2:

Thanks for the quick fix!


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 11:22:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8836/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 12:16:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 17:35:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 2:

(1 comment)

Thanks for comment, LiPenglin!

http://gerrit.cloudera.org:8080/#/c/19274/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test:

http://gerrit.cloudera.org:8080/#/c/19274/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test@541
PS1, Line 541: 2020-01-01 10:00:00
> I checked the cause of the test failure.
Thanks for looking into this! Done.



-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Nov 2022 08:09:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 1: Code-Review+1

(1 comment)

LGTM!
One minor tip for the cause of the test failure.

http://gerrit.cloudera.org:8080/#/c/19274/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test:

http://gerrit.cloudera.org:8080/#/c/19274/1/testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test@541
PS1, Line 541: 2020-01-01 10:00:00
I checked the cause of the test failure.
Table 'functional_parquet.iceberg_v2_partitioned_position_deletes' and Table 'functional_parquet.iceberg_partitioned' have different parquet metadata, the parquet data footer of 'functional_parquet.iceberg_partitioned' is 'event_time: TIMESTAMP(MICROS,/*with_timezone*/true)', maybe we need to SET 'SET TIMEZONE='Europe/Budapest' before exec the sql.



-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Thu, 24 Nov 2022 05:39:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 12:16:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11902/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Wed, 23 Nov 2022 19:04:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8835/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Nov 2022 08:09:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11906/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Nov 2022 08:28:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8837/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 12:25:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Nov 2022 11:22:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8832/


-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Comment-Date: Thu, 24 Nov 2022 01:37:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19274 )

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................

IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

If an Iceberg V2 table is partitioned, and contains delete files,
then in a query that involves runtime filters on the partition columns
return empty result set.

E.g.:

  select count(*)
  from store_sales, date_dim
  where d_date_sk = ss_sold_date_sk and d_moy=2 and d_year=1998;

In the above query store_sales is partitioned by ss_sold_date_sk which
will be filtered by runtime filters created by the JOIN. If store_sales
has delete files then the above query returns empty result set.

The problem is that we are invoking PartitionPassesFilters() on these
Iceberg tables. It is usually a no-op for Iceberg tables, as the
template tuple is NULL. But when we have virtual columns a template
tuple has been created in HdfsScanPlanNode::InitTemplateTuple. For
Iceberg tables this tempalte tuple is incomplete, i.e. it doesn't
have the partition values set. This means the filters evaluate to
false and the files are getting filtered out, hence the query
produces an empty result set.

With this patch we don't invoke PartitionPassesFilters() on Iceberg
tables, only the Iceberg-specific IcebergPartitionPassesFilters()
gets invoked. Also added DCHECKs to ensure this.

Testing:
 * e2e tests added

Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Reviewed-on: http://gerrit.cloudera.org:8080/19274
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
5 files changed, 82 insertions(+), 18 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, lipenglin@sensorsdata.cn, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19274

to look at the new patch set (#2).

Change subject: IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied
......................................................................

IMPALA-11740: Incorrect results for partitioned Iceberg V2 tables when runtime filters are applied

If an Iceberg V2 table is partitioned, and contains delete files,
then in a query that involves runtime filters on the partition columns
return empty result set.

E.g.:

  select count(*)
  from store_sales, date_dim
  where d_date_sk = ss_sold_date_sk and d_moy=2 and d_year=1998;

In the above query store_sales is partitioned by ss_sold_date_sk which
will be filtered by runtime filters created by the JOIN. If store_sales
has delete files then the above query returns empty result set.

The problem is that we are invoking PartitionPassesFilters() on these
Iceberg tables. It is usually a no-op for Iceberg tables, as the
template tuple is NULL. But when we have virtual columns a template
tuple has been created in HdfsScanPlanNode::InitTemplateTuple. For
Iceberg tables this tempalte tuple is incomplete, i.e. it doesn't
have the partition values set. This means the filters evaluate to
false and the files are getting filtered out, hence the query
produces an empty result set.

With this patch we don't invoke PartitionPassesFilters() on Iceberg
tables, only the Iceberg-specific IcebergPartitionPassesFilters()
gets invoked. Also added DCHECKs to ensure this.

Testing:
 * e2e tests added

Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes-orc.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v2-read-position-deletes.test
5 files changed, 82 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/19274/2
-- 
To view, visit http://gerrit.cloudera.org:8080/19274
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I43f3e0a4df7c1ba6d8ea61410b570d8cf7b31ad3
Gerrit-Change-Number: 19274
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>