You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org> on 2026/04/22 14:23:16 UTC
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23838
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
Instead, the partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we look up the relevant partitions by their ids in the Planner, put
them in the file descriptors that are sent to the executors.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing: ran existing e2e tests
TODO: memory and time measurements
Generated by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
15 files changed, 254 insertions(+), 80 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/3
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 3
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 11:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/23838/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23838/11//COMMIT_MSG@19
PS11, Line 19: Since the scan nodes need the partition information for execution,
: we look up the relevant partitions by their ids in the Planner, put
: them in the file descriptors that are sent to the executors.
This means that it is denormalized again when sending to executors, right? Is there a plan to make this more efficient in coordinator->executor communication too?
http://gerrit.cloudera.org:8080/#/c/23838/11/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:
http://gerrit.cloudera.org:8080/#/c/23838/11/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@220
PS11, Line 220: ByteBuffer
I think that in the long term we should replace this with arrays to save more memory. There could be followup ticket mentioned in a comment.
http://gerrit.cloudera.org:8080/#/c/23838/11/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@712
PS11, Line 712: * FlatBuffer content, regardless of how the buffer was obtained (e.g. from Thrift's
: * fast-path deserialization which may leave position > 0).
org.apache.thrift.TBaseHelper.copyBinary() could be usef for this - it is already used at same places in Impala.
https://github.com/apache/thrift/blob/2a93df80f27739ccabb5b885cb12a8dc7595ecdf/lib/java/src/org/apache/thrift/TBaseHelper.java#L272
http://gerrit.cloudera.org:8080/#/c/23838/11/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/23838/11/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1209
PS11, Line 1209: compressedBb
nit: isn't it more like compactedBb?
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 11
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 May 2026 17:11:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 10:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/10/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS10:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/10/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS10:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/10/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/10/common/thrift/CatalogObjects.thrift@681
PS10, Line 681: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 10
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 13:42:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 5:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/5/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS5:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/5/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS5:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/5/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/5/common/thrift/CatalogObjects.thrift@686
PS5, Line 686: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 5
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 May 2026 09:15:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 3:
(11 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/file-metadata-utils.h
File be/src/exec/file-metadata-utils.h:
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/file-metadata-utils.h@76
PS3, Line 76: /// Writes the Iceberg columns from FbIcebergMetadataRange into template_tuple. Updates the
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/hdfs-scan-node-base.cc@269
PS3, Line 269: file_metadata = flatbuffers::GetRoot<org::apache::impala::fb::FbFileMetadataRange>(
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/hdfs-scan-node-base.cc@914
PS3, Line 914: const FbFileMetadataRange* file_metadata = context->GetStream(0)->file_desc()->file_metadata;
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS3:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS3:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/3/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/3/common/thrift/CatalogObjects.thrift@681
PS3, Line 681: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@206
PS3, Line 206: // Partitions with their corresponding ids that are used to refer to the partition info
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1175
PS3, Line 1175: FbIcebergMetadataRange.addFirstRowId(fbb, iceMetadata.firstRowId());
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1185
PS3, Line 1185: return FbFileMetadataRange.getRootAsFbFileMetadataRange((ByteBuffer)compressedBb.flip());
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1188
PS3, Line 1188: private static int copyPartitionKeys(FlatBufferBuilder fbb, FbIcebergPartition partition) {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1205
PS3, Line 1205: partitionKeyOffsets[i] =
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 3
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Apr 2026 14:25:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 3:
(23 comments)
Thanks for the comments!
Posting this patch before the rebase to help track the changes.
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/file-metadata-utils.h
File be/src/exec/file-metadata-utils.h:
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/file-metadata-utils.h@76
PS3, Line 76: /// Writes the Iceberg columns from FbIcebergMetadataRange into template_tuple. Updates the
> line too long (93 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/hdfs-scan-node-base.cc@269
PS3, Line 269: file_metadata = flatbuffers::GetRoot<org::apache::impala::fb::FbFileMetadataRange>(
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/be/src/exec/hdfs-scan-node-base.cc@914
PS3, Line 914: const FbFileMetadataRange* file_metadata = context->GetStream(0)->file_desc()->file_metadata;
> line too long (95 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS3:
> This file is used in communication between impalad and catalogd/statestore.
Ack
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs@55
PS3, Line 55: //spec_id : ushort;
: //partition_keys : [FbIcebergPartitionTransformValue];
> This can be removed. Same goes to FbIcebergMetadataRange's comments.
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs@71
PS3, Line 71: -1
> Please comment why does it use a different default than FbIcebergMetadata.f
Actually, both this and FbIcebergMetadata.first_row_id could use -1 as default value then for this reason.
http://gerrit.cloudera.org:8080/#/c/23838/3/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/3/common/thrift/CatalogObjects.thrift@681
PS3, Line 681: 9: optional list<binary> partitions
> Changing type of field 'partitions' from list<TIcebergPartition> to list<bi
Ack
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@206
PS3, Line 206: // Partitions with their corresponding ids that are used to refer to the partition info
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@286
PS3, Line 286: partition
> Should we call normalizePartitionBuf? Or a Precondition check that partitio
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java@37
PS3, Line 37: import com.google.common.base.Preconditions;
> Unused import
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java@100
PS3, Line 100:
> nit: too much indent
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@57
PS3, Line 57: import java.nio.ByteBuffer;
> nit: misplaced import
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@82
PS3, Line 82: import java.nio.ByteBuffer;
> nit: misplaced import
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1145
PS3, Line 1145: 1
> We can use the default constructor to minimize re-allocations. At the end o
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1155
PS3, Line 1155:
> nit: indentation
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1160
PS3, Line 1160: compressedBb
> nit: we usually name these as 'compressedBb' in other methods as well, but
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1164
PS3, Line 1164: public static FbFileMetadataRange transformFileMetadata(IcebergFileDescriptor fileDesc,
> Please add a comment.
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1169
PS3, Line 1169: 1
> Same as above, we should use the default constructor.
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1175
PS3, Line 1175: FbIcebergMetadataRange.addFirstRowId(fbb, iceMetadata.firstRowId());
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1185
PS3, Line 1185: return FbFileMetadataRange.getRootAsFbFileMetadataRange((ByteBuffer)compressedBb.flip());
> line too long (93 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1188
PS3, Line 1188: private static int copyPartitionKeys(FlatBufferBuilder fbb, FbIcebergPartition partition) {
> line too long (93 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1205
PS3, Line 1205: partitionKeyOffsets[i] =
> line has trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
File fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@36
PS3, Line 36: import java.nio.ByteBuffer;
> nit: misplaced import
Done
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 3
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 May 2026 12:41:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
This change normalizes partition-related metadata out of the
file descriptors in the Catalog and the Frontend.
The partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we denormalize the file and partition metadata: we look up the
relevant partitions by their ids in the Planner, put them in the
file descriptors that are sent to the executors.
Metadata handling during scheduling and execution remains the same.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing:
- ran existing e2e tests
- extended FileMetadataLoaderTest to check the partitions
Assisted by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
16 files changed, 302 insertions(+), 146 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/15
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 15
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 15:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/15/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS15:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/15/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS15:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/15/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/15/common/thrift/CatalogObjects.thrift@681
PS15, Line 681: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 15
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 10:28:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22769/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 15
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 10:49:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 8:
(4 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/8/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS8:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/8/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS8:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/8/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/8/common/thrift/CatalogObjects.thrift@681
PS8, Line 681: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
http://gerrit.cloudera.org:8080/#/c/23838/8/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/23838/8/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@215
PS8, Line 215: FbIcebergMetadata icebergMetadata = clonedFd.getFbFileMetadata().icebergMetadata();
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 8
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 11:02:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 17: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 17
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 16:16:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 14: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 14
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jun 2026 09:13:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
This change normalizes partition-related metadata out of the
file descriptors in the Catalog and the Frontend.
The partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we denormalize the file and partition metadata: we look up the
relevant partitions by their ids in the Planner, put them in the
file descriptors that are sent to the executors.
Metadata handling during scheduling and execution remains the same.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing:
- ran existing e2e tests
- extended FileMetadataLoaderTest to check the partitions
Assisted by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
16 files changed, 303 insertions(+), 146 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/13
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 13
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 11:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/11/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS11:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/11/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS11:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/11/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/11/common/thrift/CatalogObjects.thrift@681
PS11, Line 681: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 11
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 May 2026 11:14:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 11:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22547/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 11
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 May 2026 11:33:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 14: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13567/
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 14
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jun 2026 13:44:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 13: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 13
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jun 2026 08:25:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
This change normalizes partition-related metadata out of the
file descriptors in the Catalog and the Frontend.
The partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we denormalize the file and partition metadata: we look up the
relevant partitions by their ids in the Planner, put them in the
file descriptors that are sent to the executors.
Metadata handling during scheduling and execution remains the same.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing:
- ran existing e2e tests
- extended FileMetadataLoaderTest to check the partitions
Assisted by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
16 files changed, 302 insertions(+), 146 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/16
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 16
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22208/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 3
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Apr 2026 14:45:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 8:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22484/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 8
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 11:11:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22776/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 16
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 16:19:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13590/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 17
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 16:16:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
This change normalizes partition-related metadata out of the
file descriptors in the Catalog and the Frontend.
The partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we denormalize the file and partition metadata: we look up the
relevant partitions by their ids in the Planner, put them in the
file descriptors that are sent to the executors.
Metadata handling during scheduling and execution remains the same.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing:
- ran existing e2e tests
- extended FileMetadataLoaderTest to check the partitions
Assisted by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Reviewed-on: http://gerrit.cloudera.org:8080/23838
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
16 files changed, 302 insertions(+), 146 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 18
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22490/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 10
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 14:00:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13567/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 14
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jun 2026 09:13:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 3:
(15 comments)
The change looks great overall, but a rebase is needed to provide further comments.
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs@55
PS3, Line 55: //spec_id : ushort;
: //partition_keys : [FbIcebergPartitionTransformValue];
This can be removed. Same goes to FbIcebergMetadataRange's comments.
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs@71
PS3, Line 71: -1
Please comment why does it use a different default than FbIcebergMetadata.first_row_id? I guess it's more efficient for Iceberg V1/V2 tables as first_row_id does not occupy memory (because it's always -1).
http://gerrit.cloudera.org:8080/#/c/23838/3/common/fbs/IcebergObjects.fbs@87
PS3, Line 87: From Coordinator to Catalog.
Thanks for adding these comments!
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@286
PS3, Line 286: partition
Should we call normalizePartitionBuf? Or a Precondition check that partition.position() == 0?
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@318
PS3, Line 318: needs synchronization?
If partitionList_ is constructed during fromThrift(), then it would be basically immutable after that point. Please also treat it as an immutable field (See Csaba's relevant change: 71c63e)
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java@37
PS3, Line 37: import com.google.common.base.Preconditions;
Unused import
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java@100
PS3, Line 100:
nit: too much indent
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@57
PS3, Line 57: import java.nio.ByteBuffer;
nit: misplaced import
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@82
PS3, Line 82: import java.nio.ByteBuffer;
nit: misplaced import
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java
File fe/src/main/java/org/apache/impala/util/IcebergUtil.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1145
PS3, Line 1145: 1
We can use the default constructor to minimize re-allocations. At the end of this method we return a shrinked buffer anyway.
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1155
PS3, Line 1155:
nit: indentation
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1160
PS3, Line 1160: compressedBb
nit: we usually name these as 'compressedBb' in other methods as well, but 'shrinkedBb' / 'resultBb' woule be more descriptive.
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1164
PS3, Line 1164: public static FbFileMetadataRange transformFileMetadata(IcebergFileDescriptor fileDesc,
Please add a comment.
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@1169
PS3, Line 1169: 1
Same as above, we should use the default constructor.
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
File fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java:
http://gerrit.cloudera.org:8080/#/c/23838/3/fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java@36
PS3, Line 36: import java.nio.ByteBuffer;
nit: misplaced import
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 3
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Apr 2026 09:45:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
Instead, the partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we look up the relevant partitions by their ids in the Planner, put
them in the file descriptors that are sent to the executors.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing: ran existing e2e tests
Generated by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
16 files changed, 291 insertions(+), 129 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/11
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 11
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 16: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 16
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 16:15:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 13:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/13/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS13:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/13/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS13:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/13/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/13/common/thrift/CatalogObjects.thrift@681
PS13, Line 681: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 13
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Jun 2026 14:57:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 13:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23838/12/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/12/common/thrift/CatalogObjects.thrift@679
PS12, Line 679: // Partition metadata serialized into a FlatBuffer
: // (FbIcebergPartition defined in common/fbs/IcebergObjects.fbs).
: 9: optional list<binary> partitions
> My preference would be to remove this and move to the end as 12: optional l
I think we could bump the Catalog version to V3 in CatalogService.thrift to mark the incompatibility. Created IMPALA-15060.
http://gerrit.cloudera.org:8080/#/c/23838/12/fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/23838/12/fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java@67
PS12, Line 67: ByteBuffer newDescBuf = deepCopyByteBuffer(getFbFileDescriptor().getByteBuffer());
: ByteBuffer newMetadataBuf = deepCopyByteBuffer(fbFileMetadata_.getByteBuffer());
:
: return new IcebergFi
> nit:
Done
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 13
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Jun 2026 14:55:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22676/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 13
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Jun 2026 15:18:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
Instead, the partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we look up the relevant partitions by their ids in the Planner, put
them in the file descriptors that are sent to the executors.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing: ran existing e2e tests
Generated by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
15 files changed, 238 insertions(+), 99 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/5
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 5
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22335/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 5
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 May 2026 09:36:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 10: Code-Review+1
(1 comment)
LGTM!
http://gerrit.cloudera.org:8080/#/c/23838/10/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/23838/10/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@213
PS10, Line 213: // We create a deep copy of the file descriptor for cache consistency.
: IcebergFileDescriptor clonedFd = fd.deepCopy();
Should we have a quick check to see if partId and firstRowId need modification at all?
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 10
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 May 2026 16:31:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 17: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 17
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 20:47:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 16:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23838/16/common/fbs/CatalogObjects.fbs
File common/fbs/CatalogObjects.fbs:
PS16:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/16/common/fbs/IcebergObjects.fbs
File common/fbs/IcebergObjects.fbs:
PS16:
This file is used in communication between impalad and catalogd/statestore. Please make sure impalads can still work with new/old versions of catalogd and statestore. Basically only new fields can be added and should be added at the end of a table definition.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html
http://gerrit.cloudera.org:8080/#/c/23838/16/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/23838/16/common/thrift/CatalogObjects.thrift@681
PS16, Line 681: 9: optional list<binary> partitions
Changing type of field 'partitions' from list<TIcebergPartition> to list<binary> in TIcebergContentFileStore might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 16
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2026 15:57:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
Patch Set 11: Code-Review+1
LGTM!
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 11
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 May 2026 08:56:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
Instead, the partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we look up the relevant partitions by their ids in the Planner, put
them in the file descriptors that are sent to the executors.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing: ran existing e2e tests
Generated by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
16 files changed, 268 insertions(+), 116 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/8
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 8
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/23838 )
Change subject: IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
......................................................................
IMPALA-14564: Remove redundant partition info from Iceberg file descriptors
Iceberg file descriptors used to contain information about the
partition they belong to: the spec id and the partition values.
These fields uniquely identify the partition the file belongs
to and are only dependent on the partition not the file itself.
It means that it is redundant to store these fields in each
file descriptor in the Catalog.
Instead, the partition information is stored separately in the
IcebergContentFileStore in FlatBuffers binary format. File
descriptors only store a unique id of the partition they belong to.
Since the scan nodes need the partition information for execution,
we look up the relevant partitions by their ids in the Planner, put
them in the file descriptors that are sent to the executors.
This change introduces a separate FlatBuffer schema to serialize
necessary Iceberg file metadata from the Catalog to the Frontend
and from the Frontend to the executors (in scan ranges) containing
only the relevant fields for maximum memory efficiency.
Testing: ran existing e2e tests
Generated by Copilot (Claude Sonnet 4.6)
Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
---
M be/src/exec/file-metadata-utils.cc
M be/src/exec/file-metadata-utils.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M common/fbs/CatalogObjects.fbs
M common/fbs/IcebergObjects.fbs
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergDeleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
16 files changed, 269 insertions(+), 117 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23838/10
--
To view, visit http://gerrit.cloudera.org:8080/23838
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57c2fd6f1ebb636aa9e7ca925413ca51858cbc2a
Gerrit-Change-Number: 23838
Gerrit-PatchSet: 10
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>