You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "owen6314 (via GitHub)" <gi...@apache.org> on 2025/11/09 19:34:39 UTC
[PR] Spark: Custom snapshot property from session configuration [iceberg]
owen6314 opened a new pull request, #14545:
URL: https://github.com/apache/iceberg/pull/14545
Following up on this [PR](https://github.com/apache/iceberg/pull/12999) and move it to Spark 4.0.
Adds support to allow custom snapshot properties to be specified in the Spark session configuration. This allows users to add custom snapshot properties even when running spark SQL DML such as DELETE or MERGE INTO, which was not previously supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Custom snapshot property from session configuration [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 merged PR #14545:
URL: https://github.com/apache/iceberg/pull/14545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Custom snapshot property from session configuration [iceberg]
Posted by "owen6314 (via GitHub)" <gi...@apache.org>.
owen6314 commented on code in PR #14545:
URL: https://github.com/apache/iceberg/pull/14545#discussion_r2530069858
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/TestSparkWriteConf.java:
##########
@@ -396,6 +396,37 @@ public void testSparkConfOverride() {
}
}
+ @TestTemplate
+ public void testExtraSnapshotMetadataReflectsSessionConfig() {
Review Comment:
Thanks for the comment (and sorry for the late reply). Added a E2E test `testExtraSnapshotMetadataPersistedOnWrite` for this purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Custom snapshot property from session configuration [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #14545:
URL: https://github.com/apache/iceberg/pull/14545#discussion_r2508685414
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/TestSparkWriteConf.java:
##########
@@ -396,6 +396,37 @@ public void testSparkConfOverride() {
}
}
+ @TestTemplate
+ public void testExtraSnapshotMetadataReflectsSessionConfig() {
Review Comment:
can we add a E2E test which verifies that when this is set the snapshot created actually persists the given property as snapshot property ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Custom snapshot property from session configuration [iceberg]
Posted by "yingjianwu98 (via GitHub)" <gi...@apache.org>.
yingjianwu98 commented on PR #14545:
URL: https://github.com/apache/iceberg/pull/14545#issuecomment-3538909555
+1 to this. We also have an internal use case that would greatly benefit from this enhancement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org