You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "97harsh (via GitHub)" <gi...@apache.org> on 2026/01/17 08:02:35 UTC
[PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
97harsh opened a new pull request, #15067:
URL: https://github.com/apache/iceberg/pull/15067
## Summary
- Backports branch support for the `rewrite_data_files` procedure from Spark 4.1 to Spark 4.0, 3.5, and 3.4
- This is a backport of #14964
## Changes
- Added `branch` parameter to `RewriteDataFilesProcedure` for all three Spark versions
- Added `toBranch()` method to `RewriteDataFilesSparkAction` for all three Spark versions
- Branch resolution priority: explicit parameter > table branch > main branch
- Added validation that target branch exists before rewriting
- Backported all test cases from #14964
## Test plan
- [x] Run spotless
- [x] Run tests for Spark 4.0
- [x] Run tests for Spark 3.5
- [x] Run tests for Spark 3.4 (requires Java 8/11/17)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #15067:
URL: https://github.com/apache/iceberg/pull/15067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
Posted by "97harsh (via GitHub)" <gi...@apache.org>.
97harsh commented on code in PR #15067:
URL: https://github.com/apache/iceberg/pull/15067#discussion_r2710537155
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java:
##########
@@ -109,17 +114,30 @@ public InternalRow[] call(InternalRow args) {
String sortOrderString = input.asString(SORT_ORDER_PARAM, null);
Map<String, String> options = input.asStringMap(OPTIONS_PARAM, ImmutableMap.of());
String where = input.asString(WHERE_PARAM, null);
+ String explicitBranch = input.asString(BRANCH_PARAM, null);
+
+ // Load SparkTable to extract branch from identifier
+ SparkTable sparkTable = loadSparkTable(tableIdent);
Review Comment:
It's a clean backport
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java:
##########
@@ -109,17 +114,30 @@ public InternalRow[] call(InternalRow args) {
String sortOrderString = input.asString(SORT_ORDER_PARAM, null);
Map<String, String> options = input.asStringMap(OPTIONS_PARAM, ImmutableMap.of());
String where = input.asString(WHERE_PARAM, null);
+ String explicitBranch = input.asString(BRANCH_PARAM, null);
+
+ // Load SparkTable to extract branch from identifier
+ SparkTable sparkTable = loadSparkTable(tableIdent);
Review Comment:
It's a clean backport now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
Posted by "97harsh (via GitHub)" <gi...@apache.org>.
97harsh commented on PR #15067:
URL: https://github.com/apache/iceberg/pull/15067#issuecomment-3762878303
@pvary Can you help review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15067:
URL: https://github.com/apache/iceberg/pull/15067#issuecomment-3764075554
Are these clean backports, or did you have to do any manual changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #15067:
URL: https://github.com/apache/iceberg/pull/15067#discussion_r2702902102
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java:
##########
@@ -109,17 +114,30 @@ public InternalRow[] call(InternalRow args) {
String sortOrderString = input.asString(SORT_ORDER_PARAM, null);
Map<String, String> options = input.asStringMap(OPTIONS_PARAM, ImmutableMap.of());
String where = input.asString(WHERE_PARAM, null);
+ String explicitBranch = input.asString(BRANCH_PARAM, null);
+
+ // Load SparkTable to extract branch from identifier
+ SparkTable sparkTable = loadSparkTable(tableIdent);
Review Comment:
Should we be consistent with 4.1 https://github.com/apache/iceberg/blob/main/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java#L123, only call loadSparkTable when explicitBranch == null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15067:
URL: https://github.com/apache/iceberg/pull/15067#issuecomment-3776675334
Merged to main.
Thanks @97harsh for the backport and @huaxingao and @singhpk234 for the reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Backport branch support for rewrite_data_files to Spark 4.0, 3.5, and 3.4 [iceberg]
Posted by "97harsh (via GitHub)" <gi...@apache.org>.
97harsh commented on code in PR #15067:
URL: https://github.com/apache/iceberg/pull/15067#discussion_r2710536549
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java:
##########
@@ -109,17 +114,30 @@ public InternalRow[] call(InternalRow args) {
String sortOrderString = input.asString(SORT_ORDER_PARAM, null);
Map<String, String> options = input.asStringMap(OPTIONS_PARAM, ImmutableMap.of());
String where = input.asString(WHERE_PARAM, null);
+ String explicitBranch = input.asString(BRANCH_PARAM, null);
+
+ // Load SparkTable to extract branch from identifier
+ SparkTable sparkTable = loadSparkTable(tableIdent);
Review Comment:
oh right, my bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org