You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2022/08/23 01:11:14 UTC

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Hello Tamas Mate, lipenglin@sensorsdata.cn, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to do a code review. Please visit

    http://gerrit.cloudera.org:8080/18887

to review the following change.


Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................

IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Identity-partitioned columns are not necessarily stored in the data
files. E.g. when we migrate a legacy partitioned table to Iceberg
without rewriting the data files, the partition columns won't be
present in the files.

The Parquet scanner does a few optimizations to eliminate row groups,
i.e. filtering based on stats, bloom filters, etc. When a column is
not present in the data file that has some predicate on, then it is
assumed that the whole row group doesn't pass the filtering criteria.

But for Iceberg some files might contain partition columns, while
other files doesn't, so we need to prepare the scanners to handle
such cases.

The ORC scanner doesn't have that many optimizations so it didn't
ran into this issue.

Testing:
 * e2e tests

Merge conflicts in be/src/exec/file-metadata-utils.cc due to missing
23d09638d.

Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Reviewed-on: http://gerrit.cloudera.org:8080/18605
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Reviewed-by: Tamas Mate <tm...@apache.org>
Reviewed-by: <li...@sensorsdata.cn>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
3 files changed, 142 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/18887/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 2:

Probably we should backport IMPALA-9410 as well.


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Aug 2022 13:49:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................

IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Identity-partitioned columns are not necessarily stored in the data
files. E.g. when we migrate a legacy partitioned table to Iceberg
without rewriting the data files, the partition columns won't be
present in the files.

The Parquet scanner does a few optimizations to eliminate row groups,
i.e. filtering based on stats, bloom filters, etc. When a column is
not present in the data file that has some predicate on, then it is
assumed that the whole row group doesn't pass the filtering criteria.

But for Iceberg some files might contain partition columns, while
other files doesn't, so we need to prepare the scanners to handle
such cases.

The ORC scanner doesn't have that many optimizations so it didn't
ran into this issue.

Testing:
 * e2e tests

Merge conflicts due to missing 23d09638d:
 * file-metadata-utils.cc resolves trivial conflicts
 * hdfs-parquet-scanner.cc removes usage of NeedDataInFile()

Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Reviewed-on: http://gerrit.cloudera.org:8080/18605
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Reviewed-by: Tamas Mate <tm...@apache.org>
Reviewed-by: <li...@sensorsdata.cn>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-on: http://gerrit.cloudera.org:8080/18887
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
Tested-by: Quanlong Huang <hu...@gmail.com>
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
3 files changed, 142 insertions(+), 1 deletion(-)

Approvals:
  Zoltan Borok-Nagy: Looks good to me, approved
  Quanlong Huang: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
lipenglin@sensorsdata.cn has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 2: Code-Review+1

LGTM


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Aug 2022 02:37:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 3: Verified+1

Verified by https://jenkins.impala.io/job/gerrit-verify-dryrun/8494/


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Aug 2022 12:15:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8483/


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Aug 2022 01:24:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 2: Code-Review+2

Thanks for doing the backport. Yeah, I think we need IMPALA-9410 as well.


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Aug 2022 16:58:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Tamas Mate, lipenglin@sensorsdata.cn, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18887

to look at the new patch set (#2).

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................

IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Identity-partitioned columns are not necessarily stored in the data
files. E.g. when we migrate a legacy partitioned table to Iceberg
without rewriting the data files, the partition columns won't be
present in the files.

The Parquet scanner does a few optimizations to eliminate row groups,
i.e. filtering based on stats, bloom filters, etc. When a column is
not present in the data file that has some predicate on, then it is
assumed that the whole row group doesn't pass the filtering criteria.

But for Iceberg some files might contain partition columns, while
other files doesn't, so we need to prepare the scanners to handle
such cases.

The ORC scanner doesn't have that many optimizations so it didn't
ran into this issue.

Testing:
 * e2e tests

Merge conflicts due to missing 23d09638d:
 * file-metadata-utils.cc resolves trivial conflicts
 * hdfs-parquet-scanner.cc removes usage of NeedDataInFile()

Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Reviewed-on: http://gerrit.cloudera.org:8080/18605
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Reviewed-by: Tamas Mate <tm...@apache.org>
Reviewed-by: <li...@sensorsdata.cn>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
3 files changed, 142 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/18887/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8483/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Aug 2022 01:13:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 2:

TestIcebergTable.test_migrated_tables fails:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/17356/
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/6199/

I'll try to fix them.


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Aug 2022 13:48:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR](branch-4.1.1) IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18887 )

Change subject: IMPALA-11346: Migrated partitioned Iceberg tables might return ERROR when WHERE condition is used on partition column
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: branch-4.1.1
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie706317888981f634d792fb570f3eab1ec11a4f4
Gerrit-Change-Number: 18887
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Anonymous Coward <li...@sensorsdata.cn>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Aug 2022 08:29:35 +0000
Gerrit-HasComments: No