You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Ruobing1997 (via GitHub)" <gi...@apache.org> on 2026/03/30 04:41:32 UTC
[PR] Remove Spark 2 test assumptions for write projection [iceberg]
Ruobing1997 opened a new pull request, #15823:
URL: https://github.com/apache/iceberg/pull/15823
Remove test Assumptions for Spark 2, for issue #15821
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao merged PR #15823:
URL: https://github.com/apache/iceberg/pull/15823
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on code in PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#discussion_r3012112270
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
And regarding
> why these tests only work with Spark2 and Spark4.1?
I found this issue: https://issues.apache.org/jira/browse/SPARK-51290 for allowing partial column writes
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
And regarding
> why these tests only work with Spark2 and Spark4.1?
I found this issue: https://issues.apache.org/jira/browse/SPARK-51290 for allowing partial column writes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on code in PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#discussion_r3012112270
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
And regarding
> why these tests only work with Spark2 and Spark4.1?
I found this issue: https://issues.apache.org/jira/browse/SPARK-51290 for allowing partial column writes. I noticed this was enabled in Spark 4.1. Is there any chance this could be backported to 3.5/4.0, or is it considered a new feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on code in PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#discussion_r3012103006
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
I agree with removing these tests... Seems dead code under the current logic 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#issuecomment-4159822151
Thanks @huaxingao for approving the PR. Are we good to merge it? ❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#issuecomment-4152195460
Of course! Let me take a look 👁️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#issuecomment-4160140164
Thanks @Ruobing1997 for the PR! Thanks @manuzhang for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#issuecomment-4152591838
seems the partial column write is not supported 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on code in PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#discussion_r3012112270
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
And regarding
> why these tests only work with Spark2 and Spark4.1?
I found this issue: https://issues.apache.org/jira/browse/SPARK-51290 for allowing partial column writes.
This fills default values for missing columns in DSv2 writes which seems why our tests pass on 4.1
And I have a question: I noticed this was enabled in Spark 4.1. Is there any chance this could be backported, or is it considered a new feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#issuecomment-4152190741
@Ruobing1997 I think there are more places where Spark 2 is assumed. Can you check and remove all of them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#issuecomment-4160145353
😋 Looking forward to contributing more!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#discussion_r3012077835
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
If these tests don't work on 3.4/3.5/4.0, should we remove them from those versions entirely rather than keeping dead tests? And why these tests only work with Spark2 and Spark4.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on code in PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#discussion_r3012112270
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
And regarding
> why these tests only work with Spark2 and Spark4.1?
I found this issue: https://issues.apache.org/jira/browse/SPARK-51290 for allowing partial column writes. This seems introduced in 4.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Remove Spark 2 test assumptions for write projection [iceberg]
Posted by "Ruobing1997 (via GitHub)" <gi...@apache.org>.
Ruobing1997 commented on code in PR #15823:
URL: https://github.com/apache/iceberg/pull/15823#discussion_r3012112270
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java:
##########
@@ -433,10 +427,6 @@ public void testPartitionedFanoutCreateWithTargetFileSizeViaOption2() {
Review Comment:
And regarding
> why these tests only work with Spark2 and Spark4.1?
I found this issue: https://issues.apache.org/jira/browse/SPARK-51290 for allowing partial column writes. I noticed this was enabled in Spark 4.1. Is there any chance this could be backported, or is it considered a new feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org