You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2024/09/09 17:02:39 UTC

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21761


Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. We raise an error when we find partition columns in
pre-migration data files.

Testing:
 * e2e tests added

TODO: add negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
10 files changed, 371 insertions(+), 14 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/1
-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. If none of the partition columns are included in the data
file (common case) we adjust the file-level field IDs accordingly. It is
also OK to have all the partition columns in the data files (it is not
common, but we've seen such data files). We raise an error in other
cases (some partition columns are in the data file, while others
aren't).

Testing:
 * e2e tests added
 * added negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Reviewed-on: http://gerrit.cloudera.org:8080/21761
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
13 files changed, 530 insertions(+), 15 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 12:07:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Daniel Becker, Gabor Kaszab, Mihaly Szjatinya, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21761

to look at the new patch set (#5).

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. If none of the partition columns are included in the data
file (common case) we adjust the file-level field IDs accordingly. It is
also OK to have all the partition columns in the data files (it is not
common, but we've seen such data files). We raise an error in other
cases (some partition columns are in the data file, while others
aren't).

Testing:
 * e2e tests added
 * added negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
13 files changed, 530 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/5
-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/16922/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Sep 2024 14:34:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10959/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Sep 2024 14:23:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Daniel Becker, Gabor Kaszab, Mihaly Szjatinya, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21761

to look at the new patch set (#7).

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. If none of the partition columns are included in the data
file (common case) we adjust the file-level field IDs accordingly. It is
also OK to have all the partition columns in the data files (it is not
common, but we've seen such data files). We raise an error in other
cases (some partition columns are in the data file, while others
aren't).

Testing:
 * e2e tests added
 * added negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
13 files changed, 530 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/7
-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21761

to look at the new patch set (#2).

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. We raise an error when we find partition columns in
pre-migration data files.

Testing:
 * e2e tests added
 * added negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files_orc/result_date=2024-08-26/0.orc
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files_parquet/result_date=2024-08-26/0.parquet
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M tests/query_test/test_iceberg.py
16 files changed, 438 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/2
-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/16949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 20:47:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/16904/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Sep 2024 17:34:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 11:45:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 6:

(Sorry for mess).

Checked the code and run tests. Left mostly minor comments for now.

Just one thing not directly related to this patch, but exploring the code around schema resolver I found a lot of iceberg (table format) specific ramifications across parquet and orc (file format) code.

This would definitely require better familiarity with the area,
so take it for what it is, but I wonder, would it be feasible to think in the direction of decoupling iceberg related code at some point (which would largely affect this patch among other things).


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 29 Sep 2024 14:14:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Daniel Becker, Gabor Kaszab, Mihaly Szjatinya, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21761

to look at the new patch set (#6).

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. If none of the partition columns are included in the data
file (common case) we adjust the file-level field IDs accordingly. It is
also OK to have all the partition columns in the data files (it is not
common, but we've seen such data files). We raise an error in other
cases (some partition columns are in the data file, while others
aren't).

Testing:
 * e2e tests added
 * added negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
13 files changed, 530 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/6
-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/16936/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Thu, 12 Sep 2024 16:58:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 7:

(3 comments)

Thanks for the comments, Mihaly.

About decoupling Iceberg related code: I think it's a good idea whenever we can. Currently when we are reading Iceberg tables we mostly re-use the existing classes and methods (basically the HDFS SCAN operator and the file scanners), as ultimately there are regular data files (Parquet, ORC) in an Iceberg tables. We had code for these file formats already, so in most cases it was easier to only add small enhancements to these classes than doing a large and risky refactoring.

http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h
File be/src/exec/file-metadata-utils.h:

http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h@69
PS6, Line 69: , in any other case 
> nit: ". In any other case" ?
Done


http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h@87
PS6, Line 87: Only valid to call
            :   /// for Iceberg tables.
> Can this be reflected in function's name? As it is for other Iceberg specif
Done


http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h@89
PS6, Line 89: auto
> Is there a consensus on auto usage? It doesn't seem to be used much around 
We prefer to use the actual type, but here the type is 'flatbuffers::Vector<flatbuffers::Offset<org::apache::impala::fb::FbIcebergPartitionTransformValue>>', which wouldn't be very readable. Also this is a private helper function, not meant to be used outside of this class.



-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 30 Sep 2024 16:46:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/17069/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 30 Sep 2024 17:12:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 4:

(8 comments)

Thanks for the comments!

http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc
File be/src/exec/file-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@95
PS4, Line 95: file_metadata
> should we DCHECK if 'file_metadata' is not null
Done


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@239
PS4, Line 239:   int fileFieldID = *fieldID;
> 'fieldId' could be null. Add a DCHECK?
Done


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@241
PS4, Line 241:   using namespace org::apache::impala::fb;
> I think that the code between L241-L269 is not related to the actual fieldI
I can refactor a bit if you feel strong about this, but currently it is only used by this function. Also, having complex return values then using only the first element and size wouldn't be much more readable than sourceIDs.front() and sourceIDs.back() IMO.


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@248
PS4, Line 248:   for (int i = 0; i < transforms->size(); ++i) {
> We don't need 'i' within the loop. Can this be a foreach?
Done


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@251
PS4, Line 251:             FbIcebergTransformType::FbIcebergTransformType_IDENTITY) {
> nit: too much indentation
Done


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@280
PS4, Line 280:     return Status(errorMsg);
> I think we can be more specific here with the error msg. We could let the u
Based on the field IDs only we cannot tell what's wrong exactly, it's possible that
* there are less data columns than expected
* there are more data columns than expected
* some partitions are stored, some aren't

What we know for sure is that the schema or partitioning is wrong but we cannot tell what is the exact issue, and it's also cumbersome to test all different cases. It's easier to handle them together and tell the user that something is fishy with their files. I think it should be also fairly rare to have such data files.


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/parquet/parquet-metadata-utils.cc
File be/src/exec/parquet/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/parquet/parquet-metadata-utils.cc@1022
PS4, Line 1022:       // Partition columns are not stored in file metadata, but they get field IDs
> I'm just thinking about the feasibility of the comment I left in FileMetada
I'm not sure I if get this, but collecting the children's field IDs, then doing the validation check, then adjusting field IDs seem to be more complicated to me than doing this check here. It is also checked only once, when we iterated over the top-level schema elements, and we just adjust a single integer 'fieldID'.

Please note that we also fill out 'schema_node_to_field_id_' during the process.

"So we come to this part of the code for the children of complex types, and only if this is a migrated, partitioned Iceberg table, right?"
We enter the if statement if

* table is migrated (otherwise GenerateFieldIDs() is not called)
* we just iterated over the top-level columns (current == &schema_)
* there are complex types to process (!nodes.empty())


http://gerrit.cloudera.org:8080/#/c/21761/4/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:

http://gerrit.cloudera.org:8080/#/c/21761/4/tests/query_test/test_iceberg.py@281
PS4, Line 281:             id INT, 
> line has trailing whitespace
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 20:20:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10942/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Sep 2024 14:01:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 8: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 17:18:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 4:

(7 comments)

Thanks for fixing this, Zoltan!

http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc
File be/src/exec/file-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@95
PS4, Line 95: file_metadata
should we DCHECK if 'file_metadata' is not null


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@239
PS4, Line 239:   int fileFieldID = *fieldID;
'fieldId' could be null. Add a DCHECK?


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@241
PS4, Line 241:   using namespace org::apache::impala::fb;
I think that the code between L241-L269 is not related to the actual fieldId you gave to this function and the same code runs multiple times but would give the same answer. I think we could extract it somewhere else and we could pass this function only the result of that code block.
As I see the actual sourceId vector isn't relevant either after L271, just the first, last elements and the size. In fact once we verified that they are consecutive numbers, only the first element and the size matters.


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@248
PS4, Line 248:   for (int i = 0; i < transforms->size(); ++i) {
We don't need 'i' within the loop. Can this be a foreach?


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@251
PS4, Line 251:             FbIcebergTransformType::FbIcebergTransformType_IDENTITY) {
nit: too much indentation


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/file-metadata-utils.cc@280
PS4, Line 280:     return Status(errorMsg);
I think we can be more specific here with the error msg. We could let the user know that only the subset of the partitions are written into the data file.


http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/parquet/parquet-metadata-utils.cc
File be/src/exec/parquet/parquet-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/21761/4/be/src/exec/parquet/parquet-metadata-utils.cc@1022
PS4, Line 1022:       // Partition columns are not stored in file metadata, but they get field IDs
I'm just thinking about the feasibility of the comment I left in FileMetadatUtils. So we come to this part of the code for the children of complex types, and only if this is a migrated, partitioned Iceberg table, right?
What if we collect such fields first, do the validations for the Partition fields once, and then adjust the fildIds?



-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Fri, 13 Sep 2024 12:07:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 3:

(2 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python2 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with virtualenv installed.

http://gerrit.cloudera.org:8080/#/c/21761/3/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:

http://gerrit.cloudera.org:8080/#/c/21761/3/tests/query_test/test_iceberg.py@281
PS3, Line 281:  
flake8: W291 trailing whitespace


http://gerrit.cloudera.org:8080/#/c/21761/3/tests/query_test/test_iceberg.py@281
PS3, Line 281:             id INT, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Sep 2024 16:28:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10948/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 20:20:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10948/


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 22:25:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 6:

(3 comments)

Checked the code and run tests. Left mostly minor comments for now.
 
Just one thing not directly related to this patch, but exploring the code around schema resolver I found a lot of iceberg (table format) specific ramifications across parquet and orc (file format) code.

This would definitely require better familiarity with the area, so take it for what it is, but I wonder, would it be feasible to think in the direction of decoupling iceberg related code at some point? (which would largely affect this patch among other things).

http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h
File be/src/exec/file-metadata-utils.h:

http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h@69
PS6, Line 69: , in any other cases
nit: ". In any other case" ?


http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h@87
PS6, Line 87: Only valid to call
            :   /// for Iceberg tables.
Can this be reflected in function's name? As it is for other Iceberg specific functions around.


http://gerrit.cloudera.org:8080/#/c/21761/6/be/src/exec/file-metadata-utils.h@89
PS6, Line 89: auto
Is there a consensus on auto usage? It doesn't seem to be used much around these files.



-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 29 Sep 2024 14:08:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10991/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 12:07:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 7:

Makes sense.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 30 Sep 2024 16:53:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Sep 2024 19:29:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/16979/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Sep 2024 14:55:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 6:

> (3 comments)
 > 
 > Checked the code and run tests. Left mostly minor comments for now.
 > 
 > Just one thing not directly related to this patch, but exploring
 > the code around schema resolver I found a lot of iceberg (table
 > format) specific ramifications across parquet and orc (file format)
 > code.
 > 
 > This would definitely require better familiarity with the area, so
 > take it for what it is, but I wonder, would it be feasible to think
 > in the direction of decoupling iceberg related code at some point?
 > (which would largely affect this patch among other things).


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 29 Sep 2024 14:11:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10942/


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Sep 2024 16:05:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 4:

(2 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python2 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with virtualenv installed.

http://gerrit.cloudera.org:8080/#/c/21761/4/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:

http://gerrit.cloudera.org:8080/#/c/21761/4/tests/query_test/test_iceberg.py@281
PS4, Line 281:  
flake8: W291 trailing whitespace


http://gerrit.cloudera.org:8080/#/c/21761/4/tests/query_test/test_iceberg.py@281
PS4, Line 281:             id INT, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Sep 2024 16:32:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21761

to look at the new patch set (#3).

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. If none of the partition columns are included in the data
file (common case) we adjust the file-level field IDs accordingly. It is
also OK to have all the partition columns in the data files (it is not
common, but we've seen such data files). We raise an error in other
cases (some partition columns are in the data file, while others
aren't).

Testing:
 * e2e tests added
 * added negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
M testdata/datasets/functional/functional_schema_template.sql
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
14 files changed, 567 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/3
-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21761

to look at the new patch set (#4).

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................

IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Schema resolution doesn't work correctly for migrated partitioned
Iceberg tables that have complex types. When we face a Parquet/ORC file
in an Iceberg table that doesn't have field IDs in the file metadata, we
assume that it is an old data file before migration, and the schema is
the very first one, hence we can mimic Iceberg's field ID generation to
assign field IDs to the file schema elements.

This process didn't take the partition columns into account. Partition
columns are not part of the data file but they still get field IDs. This
only matters when there are complex types in the table, as partition
columns are always the last columns in legacy Hive tables, and field IDs
are assigned via a "BFS-like" traversal. I.e. if there are only primitive
types in the table we don't have any problems, but the children of
complex types columns are assigned incorrectly.

This patch fixes field ID generation by taking the number of partitions
into account. If none of the partition columns are included in the data
file (common case) we adjust the file-level field IDs accordingly. It is
also OK to have all the partition columns in the data files (it is not
common, but we've seen such data files). We raise an error in other
cases (some partition columns are in the data file, while others
aren't).

Testing:
 * e2e tests added
 * added negative tests

Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/orc/hdfs-orc-scanner.cc
M be/src/exec/orc/orc-metadata-utils.cc
M be/src/exec/orc/orc-metadata-utils.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-metadata-utils.h
A testdata/migrated_iceberg/README
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.orc
A testdata/migrated_iceberg/complextypes_and_partition_columns_in_data_files.parquet
A testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-table-field-id-resolution-complex.test
M tests/query_test/test_iceberg.py
13 files changed, 529 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/21761/4
-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21761 )

Change subject: IMPALA-13364: Schema resolution doesn't work for migrated partitioned Iceberg tables that have complex types
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/16937/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie32952021b63d6b55b8820489e434bfc2a91580b
Gerrit-Change-Number: 21761
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Comment-Date: Thu, 12 Sep 2024 17:03:52 +0000
Gerrit-HasComments: No