You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dhruv-pratap (via GitHub)" <gi...@apache.org> on 2025/08/22 21:09:47 UTC
[PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
dhruv-pratap opened a new pull request, #13903:
URL: https://github.com/apache/iceberg/pull/13903
## Summary
This PR introduces distributed scanning capabilities for the `PartitionsTable` metadata table to improve performance when querying tables with many manifest files.
## Why is this needed?
Today partitions metadata table is processes in a single task/thread which can be limiting for engines like Spark when scanning partitions metadata of very large table with very large number of manifests. Sometimes also leading to the process (Spark driver) going OOM. This PR enables parallel processing of partition metadata scanning for large tables with many manifest files, significantly reducing query latency for partition metadata table operations.
## Key Changes
- **New Planning Modes**: Added `LOCAL`, `DISTRIBUTED`, and `AUTO` modes via `METADATA_PLANNING_MODE` table property
- **Auto-switching**: Automatically uses distributed scanning when manifest count exceeds configurable threshold (default: 10)
- **Enhanced PartitionsTable**: Implements `DistributedPartitionsScan` for parallel manifest processing
- **Comprehensive Testing**: Added tests for core functionality and Spark integration (v3.5, v4.0)
- **Backward Compatibility**: Existing behavior preserved with `AUTO` mode as default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "dhruv-pratap (via GitHub)" <gi...@apache.org>.
dhruv-pratap commented on code in PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#discussion_r2303968811
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -244,6 +244,12 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ public static final String METADATA_PLANNING_MODE = "read.metadata-planning-mode";
Review Comment:
I think this ties into @RussellSpitzer 's point he's made above. Let me amend the PR to remove the "Planning mode" complexity and simply swap the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#discussion_r2301596538
##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -331,4 +383,128 @@ private static PartitionData toPartitionData(StructLike key, Types.StructType ke
return keyTemplate.copyFor(key);
}
}
+
+ /** Distributed scan for partitions table that creates multiple tasks for processing manifests. */
+ private static class DistributedPartitionsScan extends BaseMetadataTableScan {
Review Comment:
I feel like we should have logic for distributed scans in the `spark` module where needed instead of here, and potentially reusing the files table + group by like Russell mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#issuecomment-3221135831
I think this is a great idea, but this is a relatively heavy approach to implementation. I think we really don't need to do much more than swapping the PartitionsTable implementation with the "ManifestEntries" table implementation (+ a group and distinct?)
I also think this complexity is probably not needed
```
New Planning Modes: Added LOCAL, DISTRIBUTED, and AUTO modes via METADATA_PLANNING_MODE table property
Auto-switching: Automatically uses distributed scanning when manifest count exceeds configurable threshold (default: 10)
Enhanced PartitionsTable: Implements DistributedPartitionsScan for parallel manifest processing
Comprehensive Testing: Added tests for core functionality and Spark integration (v3.5, v4.0)
Backward Compatibility: Existing behavior preserved with AUTO mode as default
```
We can probably just swap out the implementation and be done with it. Almost everyone I know who relies on the "partitions" table actually does a "SELECT * FROM FILES/ENTRIES Group by Partition" anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "dhruv-pratap (via GitHub)" <gi...@apache.org>.
dhruv-pratap commented on code in PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#discussion_r2303976780
##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -331,4 +383,128 @@ private static PartitionData toPartitionData(StructLike key, Types.StructType ke
return keyTemplate.copyFor(key);
}
}
+
+ /** Distributed scan for partitions table that creates multiple tasks for processing manifests. */
+ private static class DistributedPartitionsScan extends BaseMetadataTableScan {
Review Comment:
I wanted the implementation to be consistent with the other MetadataTable scans that are distributed - files, manifests - and those are part of the core module at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#issuecomment-3237152390
> But if there’s consensus, I’m more than happy to fully swap out the implementation. That would definitely simplify the solution and likely eliminate the need for the parameterized test coverage introduced by the new planning modes.
I'm obviously in favor :) I don't think the old implementation should be here at all :)
We can send an email out on the dev list but I think we'll find pretty much everyone is using files/entries instead of "partitions" because of performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#discussion_r2301596538
##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -331,4 +383,128 @@ private static PartitionData toPartitionData(StructLike key, Types.StructType ke
return keyTemplate.copyFor(key);
}
}
+
+ /** Distributed scan for partitions table that creates multiple tasks for processing manifests. */
+ private static class DistributedPartitionsScan extends BaseMetadataTableScan {
Review Comment:
I feel like we should model the distributed partitions table scan after the distributed scans for other metadata tables, and have logic in the `spark` module where needed instead of here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "dhruv-pratap (via GitHub)" <gi...@apache.org>.
dhruv-pratap commented on PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#issuecomment-3224905636
> I think this is a great idea, but this is a relatively heavy approach to implementation. I think we really don't need to do much more than swapping the PartitionsTable implementation with the "ManifestEntries" table implementation (+ a group and distinct?)
>
> I also think this complexity is probably not needed
>
> ```
> New Planning Modes: Added LOCAL, DISTRIBUTED, and AUTO modes via METADATA_PLANNING_MODE table property
> Auto-switching: Automatically uses distributed scanning when manifest count exceeds configurable threshold (default: 10)
> Enhanced PartitionsTable: Implements DistributedPartitionsScan for parallel manifest processing
> Comprehensive Testing: Added tests for core functionality and Spark integration (v3.5, v4.0)
> Backward Compatibility: Existing behavior preserved with AUTO mode as default
> ```
>
> We can probably just swap out the implementation and be done with it. Almost everyone I know who relies on the "partitions" table actually does a "SELECT * FROM FILES/ENTRIES Group by Partition" anyway
@RussellSpitzer I agree—that’s the same workaround we recommend to our users internally at Netflix as well. The reason I initially went with a more flexible approach was because I wasn’t sure how the community would feel about completely replacing the original implementation. I wanted to preserve backward compatibility just in case.
But if there’s consensus, I’m more than happy to fully swap out the implementation. That would definitely simplify the solution and likely eliminate the need for the parameterized test coverage introduced by the new planning modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on code in PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#discussion_r2301560986
##########
core/src/main/java/org/apache/iceberg/TableProperties.java:
##########
@@ -244,6 +244,12 @@ private TableProperties() {}
public static final String DELETE_PLANNING_MODE = "read.delete-planning-mode";
public static final String PLANNING_MODE_DEFAULT = PlanningMode.AUTO.modeName();
+ public static final String METADATA_PLANNING_MODE = "read.metadata-planning-mode";
Review Comment:
Is this needed? The existing distributed scans in Spark don't respect this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#issuecomment-3344438545
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#discussion_r2298705218
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestSparkDistributedPartitionsTable.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestTemplate;
+
+/**
+ * Integration tests for distributed PartitionsTable scanning in Spark.
+ *
+ * <p>Tests the complete flow from SQL queries like: SELECT max(max_utc_date) FROM table__partitions
+ *
+ * <p>Down to the distributed partition metadata scanning implementation.
+ */
+public class TestSparkDistributedPartitionsTable extends CatalogTestBase {
Review Comment:
Why do we need Spark Specific tests here? Shouldn't we be covered by existing test suites in MetadataTable Scans?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13903:
URL: https://github.com/apache/iceberg/pull/13903#issuecomment-3374710249
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support Distributed Scan For Partitions Metadata Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13903: Core: Support Distributed Scan For Partitions Metadata Table
URL: https://github.com/apache/iceberg/pull/13903
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org