You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rdblue (via GitHub)" <gi...@apache.org> on 2025/01/17 03:21:46 UTC
[PR] Spark 3.4: Backport support for default values [iceberg]
rdblue opened a new pull request, #11987:
URL: https://github.com/apache/iceberg/pull/11987
This backports support for default values from 3.5.
Each PR is backported as a separate commit: #11299, #11803, #11811, #11815, and #11832.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Backport support for default values [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #11987:
URL: https://github.com/apache/iceberg/pull/11987#issuecomment-2597462630
Thanks @rdblue!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Backport support for default values [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #11987:
URL: https://github.com/apache/iceberg/pull/11987#discussion_r1919504716
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReader.java:
##########
@@ -62,8 +62,7 @@ public void writeAndValidateRepeatingRecords() throws IOException {
private void writeAndValidateRecords(Schema schema, Iterable<InternalRow> expected)
throws IOException {
- final File testFile = temp.newFile();
- Assert.assertTrue("Delete should succeed", testFile.delete());
+ final File testFile = temp.resolve("test").toFile();
Review Comment:
This test suite also hit issues with JUnit 4 so I ported all the `AvroDataTest` suites to JUnit 5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Backport support for default values [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #11987:
URL: https://github.com/apache/iceberg/pull/11987#discussion_r1919575006
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkAvroReader.java:
##########
@@ -20,45 +20,53 @@
import static org.apache.iceberg.spark.data.TestHelpers.assertEqualsUnsafe;
-import java.io.File;
import java.io.IOException;
import java.util.List;
import org.apache.avro.generic.GenericData.Record;
-import org.apache.iceberg.Files;
import org.apache.iceberg.Schema;
import org.apache.iceberg.avro.Avro;
import org.apache.iceberg.avro.AvroIterable;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.OutputFile;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.spark.sql.catalyst.InternalRow;
-import org.junit.Assert;
public class TestSparkAvroReader extends AvroDataTest {
@Override
protected void writeAndValidate(Schema schema) throws IOException {
- List<Record> expected = RandomData.generateList(schema, 100, 0L);
+ writeAndValidate(schema, schema);
+ }
+
+ @Override
+ protected void writeAndValidate(Schema writeSchema, Schema expectedSchema) throws IOException {
+ List<Record> expected = RandomData.generateList(writeSchema, 100, 0L);
- File testFile = temp.newFile();
- Assert.assertTrue("Delete should succeed", testFile.delete());
+ OutputFile outputFile = new InMemoryOutputFile();
Review Comment:
Yeah I compared it side by side with the original, I think the changes made are reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Backport support for default values [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #11987:
URL: https://github.com/apache/iceberg/pull/11987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Backport support for default values [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #11987:
URL: https://github.com/apache/iceberg/pull/11987#discussion_r1919504140
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkAvroReader.java:
##########
@@ -20,45 +20,53 @@
import static org.apache.iceberg.spark.data.TestHelpers.assertEqualsUnsafe;
-import java.io.File;
import java.io.IOException;
import java.util.List;
import org.apache.avro.generic.GenericData.Record;
-import org.apache.iceberg.Files;
import org.apache.iceberg.Schema;
import org.apache.iceberg.avro.Avro;
import org.apache.iceberg.avro.AvroIterable;
+import org.apache.iceberg.inmemory.InMemoryOutputFile;
import org.apache.iceberg.io.FileAppender;
+import org.apache.iceberg.io.OutputFile;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.spark.sql.catalyst.InternalRow;
-import org.junit.Assert;
public class TestSparkAvroReader extends AvroDataTest {
@Override
protected void writeAndValidate(Schema schema) throws IOException {
- List<Record> expected = RandomData.generateList(schema, 100, 0L);
+ writeAndValidate(schema, schema);
+ }
+
+ @Override
+ protected void writeAndValidate(Schema writeSchema, Schema expectedSchema) throws IOException {
+ List<Record> expected = RandomData.generateList(writeSchema, 100, 0L);
- File testFile = temp.newFile();
- Assert.assertTrue("Delete should succeed", testFile.delete());
+ OutputFile outputFile = new InMemoryOutputFile();
Review Comment:
Spark 3.5 tests have been ported to JUnit 5, but 3.4 and 3.3 had not been. This required a few changes that I tried to keep to a minimum. This is one example, where the base class uses a Unit 4 `TemporaryFolder` that is not initialized for JUnit 5 parameterized tests.
Using `InMemoryOutputFile` here and in a couple other test suites avoided the need to port tests in #11811 to JUnit 5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org