You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "krisnaru (via GitHub)" <gi...@apache.org> on 2025/10/16 21:29:37 UTC
[PR] Spark 4.0: CopyTable support for multiple source and destination prefixes [iceberg]
krisnaru opened a new pull request, #14355:
URL: https://github.com/apache/iceberg/pull/14355
### Overview
Enhanced Apache Iceberg's RewriteTablePathSparkAction to support multiple source-target prefix pairs through a fluent chaining API, enabling complex table migration scenarios with hierarchical path mappings.
### Problem Statement
The original implementation only supported a single source-target prefix pair, limiting users to simple one-to-one path transformations. This was insufficient for:
Multi-cloud migrations with different storage systems
Complex data reorganization with multiple path hierarchies
Cross-environment moves requiring multiple prefix mapping
Multiple hadoop clusters support
Usage
// Before: Single prefix only
.rewriteLocationPrefix(sourcePrefix, targetPrefix)
// After: Chainable multiple prefixes
.rewriteLocationPrefix("s3://old-bucket/", "s3://new-bucket/")
.rewriteLocationPrefix("hdfs://cluster/", "s3://data-lake/")
.rewriteLocationPrefix("/tmp/", "s3://staging/")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: CopyTable support for multiple source and destination prefixes [iceberg]
Posted by "Rajeev-01 (via GitHub)" <gi...@apache.org>.
Rajeev-01 commented on code in PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#discussion_r2447561532
##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -608,6 +610,43 @@ private static PositionDelete newPositionDeleteRecord(
return delete;
}
+ /**
+ * Lookup the longest matching prefix mapping for a given path.
+ *
+ * @param path the path to find a prefix mapping for
+ * @param prefixMappings map of source prefix to target prefix mappings
+ * @return the Map.Entry with the longest matching source prefix, or null if no match found
+ */
+ public static Map.Entry<String, String> lookupPrefixMappings(
+ String path, Map<String, String> prefixMappings) {
+ if (prefixMappings == null || prefixMappings.isEmpty() || path == null) {
+ return null;
+ }
+
+ // Sort entries by key length in descending order to find the longest matching prefix
+ return prefixMappings.entrySet().stream()
+ .filter(entry -> path.startsWith(entry.getKey()))
+ .max(java.util.Comparator.comparing(entry -> entry.getKey().length()))
+ .orElse(null);
Review Comment:
Hey @krisnaru I didn't get why are we sorting by length of the paths that we sent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: CopyTable support for multiple source and destination prefixes [iceberg]
Posted by "krisnaru (via GitHub)" <gi...@apache.org>.
krisnaru commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-3746870540
@anuragmantri can you help get eyes on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: CopyTable support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-3590734460
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: CopyTable support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-3560743705
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-3792134376
I think this is a useful feature for users who have moved the data / metadata locations during the life of a table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-5533883761
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-5627450406
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "krisnaru (via GitHub)" <gi...@apache.org>.
krisnaru commented on code in PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#discussion_r3186041706
##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -608,6 +610,43 @@ private static PositionDelete newPositionDeleteRecord(
return delete;
}
+ /**
+ * Lookup the longest matching prefix mapping for a given path.
+ *
+ * @param path the path to find a prefix mapping for
+ * @param prefixMappings map of source prefix to target prefix mappings
+ * @return the Map.Entry with the longest matching source prefix, or null if no match found
+ */
+ public static Map.Entry<String, String> lookupPrefixMappings(
+ String path, Map<String, String> prefixMappings) {
+ if (prefixMappings == null || prefixMappings.isEmpty() || path == null) {
+ return null;
+ }
+
+ // Sort entries by key length in descending order to find the longest matching prefix
+ return prefixMappings.entrySet().stream()
+ .filter(entry -> path.startsWith(entry.getKey()))
+ .max(java.util.Comparator.comparing(entry -> entry.getKey().length()))
+ .orElse(null);
Review Comment:
If you have multiple prefix mappings like:
```
s3://bucket/warehouse/ → s3://new-bucket/warehouse/
s3://bucket/warehouse/db/tbl/ → s3://other-bucket/data/
```
And a file path is s3://bucket/warehouse/db/tbl/data.parquet, both prefixes match (both are valid startsWith matches). By picking the longest matching prefix, you ensure the most specific mapping wins — the
file gets rewritten to s3://other-bucket/data/data.parquet rather than s3://new-bucket/warehouse/db/tbl/data.parquet.
Without the length-based sorting, the result would be non-deterministic (depends on map iteration order) and could apply the wrong, less-specific prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "Priyadarshini-Mitra (via GitHub)" <gi...@apache.org>.
Priyadarshini-Mitra commented on code in PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#discussion_r3329717704
##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -608,6 +610,43 @@ private static PositionDelete newPositionDeleteRecord(
return delete;
}
+ /**
+ * Lookup the longest matching prefix mapping for a given path.
+ *
+ * @param path the path to find a prefix mapping for
+ * @param prefixMappings map of source prefix to target prefix mappings
+ * @return the Map.Entry with the longest matching source prefix, or null if no match found
+ */
+ public static Map.Entry<String, String> lookupPrefixMappings(
+ String path, Map<String, String> prefixMappings) {
+ if (prefixMappings == null || prefixMappings.isEmpty() || path == null) {
+ return null;
+ }
+
+ // Sort entries by key length in descending order to find the longest matching prefix
+ return prefixMappings.entrySet().stream()
+ .filter(entry -> path.startsWith(entry.getKey()))
+ .max(java.util.Comparator.comparing(entry -> entry.getKey().length()))
+ .orElse(null);
Review Comment:
I think the idea is to make it deterministic in some way so users know which one would be chosen when there is a common prefix we would pick the longest prefix in that case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-4861204599
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#issuecomment-4375642765
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14355: Core, Spark: RewriteTablePath support for multiple source and destination prefixes
URL: https://github.com/apache/iceberg/pull/14355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: CopyTable support for multiple source and destination prefixes [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14355: Spark 4.0: CopyTable support for multiple source and destination prefixes
URL: https://github.com/apache/iceberg/pull/14355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "Priyadarshini-Mitra (via GitHub)" <gi...@apache.org>.
Priyadarshini-Mitra commented on code in PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#discussion_r3329717704
##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -608,6 +610,43 @@ private static PositionDelete newPositionDeleteRecord(
return delete;
}
+ /**
+ * Lookup the longest matching prefix mapping for a given path.
+ *
+ * @param path the path to find a prefix mapping for
+ * @param prefixMappings map of source prefix to target prefix mappings
+ * @return the Map.Entry with the longest matching source prefix, or null if no match found
+ */
+ public static Map.Entry<String, String> lookupPrefixMappings(
+ String path, Map<String, String> prefixMappings) {
+ if (prefixMappings == null || prefixMappings.isEmpty() || path == null) {
+ return null;
+ }
+
+ // Sort entries by key length in descending order to find the longest matching prefix
+ return prefixMappings.entrySet().stream()
+ .filter(entry -> path.startsWith(entry.getKey()))
+ .max(java.util.Comparator.comparing(entry -> entry.getKey().length()))
+ .orElse(null);
Review Comment:
I think the idea is to make it deterministic in some way so users know which one would be chosen when there is a common prefix we would pick the longest common prefix in that case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 4.0: RewriteTablePath support for multiple source and destination prefixes [iceberg]
Posted by "Priyadarshini-Mitra (via GitHub)" <gi...@apache.org>.
Priyadarshini-Mitra commented on code in PR #14355:
URL: https://github.com/apache/iceberg/pull/14355#discussion_r3329717704
##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -608,6 +610,43 @@ private static PositionDelete newPositionDeleteRecord(
return delete;
}
+ /**
+ * Lookup the longest matching prefix mapping for a given path.
+ *
+ * @param path the path to find a prefix mapping for
+ * @param prefixMappings map of source prefix to target prefix mappings
+ * @return the Map.Entry with the longest matching source prefix, or null if no match found
+ */
+ public static Map.Entry<String, String> lookupPrefixMappings(
+ String path, Map<String, String> prefixMappings) {
+ if (prefixMappings == null || prefixMappings.isEmpty() || path == null) {
+ return null;
+ }
+
+ // Sort entries by key length in descending order to find the longest matching prefix
+ return prefixMappings.entrySet().stream()
+ .filter(entry -> path.startsWith(entry.getKey()))
+ .max(java.util.Comparator.comparing(entry -> entry.getKey().length()))
+ .orElse(null);
Review Comment:
I think the idea is to make it deterministic in some way so users know which one would be chosen when there is a common prefix we would pick the longest prefix in that case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org