You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tomtongue (via GitHub)" <gi...@apache.org> on 2025/05/13 08:52:55 UTC
[PR] Spark 3.4: Migrate tests in spark.data including refactoring Spark 3.5 tests [iceberg]
tomtongue opened a new pull request, #13038:
URL: https://github.com/apache/iceberg/pull/13038
*Migrate Spark 3.4 tests based on JUnit 4 to Junit5 with AssertJ style. This is related to https://github.com/apache/iceberg/issues/7160*
This PR migrates the following tests in `spark.data` to JUnit 5 with AssertJ:
`data.parquet.vectorized`
* `TestParquetDictionaryFallbackToPlainEncodingVectorizedReads`
* `TestParquetVectorizedReads`
`data`
* `GenericsHelpers` including refactoring Spark 3.5 tests
* `TestHelpers` including refectoring Spark 3.5 tests
* `TestOrcWrite`
* `TestParquetAvroReader`
* `TestParquetAvroWriter`
* `TestSparkAvroEnums`
* `TestSparkDateTimes`
* `TestSparkOrcReadMetadataColumns`
* `TestSparkParquetReadMetadataColumns`
* `TestSparkParquetWriter` including the backport from https://github.com/apache/iceberg/pull/10149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Migrate tests in spark.data including refactoring Spark 3.5 tests [iceberg]
Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #13038:
URL: https://github.com/apache/iceberg/pull/13038#discussion_r2086476697
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetWriter.java:
##########
@@ -18,27 +18,39 @@
*/
package org.apache.iceberg.spark.data;
+import static org.apache.iceberg.TableProperties.PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX;
+import static org.apache.iceberg.TableProperties.PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX;
import static org.apache.iceberg.types.Types.NestedField.optional;
import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
import java.io.File;
import java.io.IOException;
+import java.lang.reflect.Field;
+import java.nio.file.Path;
import java.util.Iterator;
import org.apache.iceberg.Files;
import org.apache.iceberg.Schema;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.io.FileAppender;
import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.parquet.ParquetSchemaUtil;
import org.apache.iceberg.spark.SparkSchemaUtil;
import org.apache.iceberg.types.Types;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.schema.MessageType;
import org.apache.spark.sql.catalyst.InternalRow;
-import org.junit.Assert;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
public class TestSparkParquetWriter {
- @Rule public TemporaryFolder temp = new TemporaryFolder();
+ @TempDir private Path temp;
+
+ public static final Schema SCHEMA =
Review Comment:
Thanks for checking this part. Let me apply this for Spark 3.5 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Migrate tests in spark.data including refactoring Spark 3.5 tests [iceberg]
Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on PR #13038:
URL: https://github.com/apache/iceberg/pull/13038#issuecomment-2875975267
@nastra Thanks for the review! the `spark.source` package is only left, so I believe one or two PRs will complete the Spark migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Migrate tests in spark.data including refactoring Spark 3.5 tests [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #13038:
URL: https://github.com/apache/iceberg/pull/13038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Migrate tests in spark.data including refactoring Spark 3.5 tests [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13038:
URL: https://github.com/apache/iceberg/pull/13038#discussion_r2086440502
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetWriter.java:
##########
@@ -18,27 +18,39 @@
*/
package org.apache.iceberg.spark.data;
+import static org.apache.iceberg.TableProperties.PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX;
+import static org.apache.iceberg.TableProperties.PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX;
import static org.apache.iceberg.types.Types.NestedField.optional;
import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
import java.io.File;
import java.io.IOException;
+import java.lang.reflect.Field;
+import java.nio.file.Path;
import java.util.Iterator;
import org.apache.iceberg.Files;
import org.apache.iceberg.Schema;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.io.FileAppender;
import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.parquet.ParquetSchemaUtil;
import org.apache.iceberg.spark.SparkSchemaUtil;
import org.apache.iceberg.types.Types;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.schema.MessageType;
import org.apache.spark.sql.catalyst.InternalRow;
-import org.junit.Assert;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
public class TestSparkParquetWriter {
- @Rule public TemporaryFolder temp = new TemporaryFolder();
+ @TempDir private Path temp;
+
+ public static final Schema SCHEMA =
Review Comment:
```suggestion
private static final Schema SCHEMA =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Migrate tests in spark.data including refactoring Spark 3.5 tests [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #13038:
URL: https://github.com/apache/iceberg/pull/13038#discussion_r2086462114
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/data/TestHelpers.java:
##########
@@ -561,21 +556,20 @@ private static void assertEqualsLists(
private static void assertEqualsMaps(
String prefix, Types.MapType type, MapData expected, Map<?, ?> actual) {
if (expected == null || actual == null) {
- Assert.assertEquals(prefix, expected, actual);
+ assertThat(actual).as(prefix).isEqualTo(expected);
} else {
Type keyType = type.keyType();
Type valueType = type.valueType();
ArrayData expectedKeyArray = expected.keyArray();
ArrayData expectedValueArray = expected.valueArray();
- Assert.assertEquals(prefix + " length", expected.numElements(), actual.size());
+ assertThat(actual).as(prefix + " length").hasSize(expectedKeyArray.numElements());
for (int e = 0; e < expected.numElements(); ++e) {
Object expectedKey = getValue(expectedKeyArray, e, keyType);
Object actualValue = actual.get(expectedKey);
if (actualValue == null) {
- Assert.assertEquals(
- prefix + ".key=" + expectedKey + " has null",
- true,
- expected.valueArray().isNullAt(e));
+ assertThat(true)
Review Comment:
this assertions looks a bit weird. I would suggest to rewrite this to `assertThat(expected.valueArray().isNullAt(e)).as("%s.key=%s has null", prefix, expectedKey).isTrue();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.4: Migrate tests in spark.data including refactoring Spark 3.5 tests [iceberg]
Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on code in PR #13038:
URL: https://github.com/apache/iceberg/pull/13038#discussion_r2086478536
##########
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/data/TestHelpers.java:
##########
@@ -561,21 +556,20 @@ private static void assertEqualsLists(
private static void assertEqualsMaps(
String prefix, Types.MapType type, MapData expected, Map<?, ?> actual) {
if (expected == null || actual == null) {
- Assert.assertEquals(prefix, expected, actual);
+ assertThat(actual).as(prefix).isEqualTo(expected);
} else {
Type keyType = type.keyType();
Type valueType = type.valueType();
ArrayData expectedKeyArray = expected.keyArray();
ArrayData expectedValueArray = expected.valueArray();
- Assert.assertEquals(prefix + " length", expected.numElements(), actual.size());
+ assertThat(actual).as(prefix + " length").hasSize(expectedKeyArray.numElements());
for (int e = 0; e < expected.numElements(); ++e) {
Object expectedKey = getValue(expectedKeyArray, e, keyType);
Object actualValue = actual.get(expectedKey);
if (actualValue == null) {
- Assert.assertEquals(
- prefix + ".key=" + expectedKey + " has null",
- true,
- expected.valueArray().isNullAt(e));
+ assertThat(true)
Review Comment:
I missed this part, thanks for the suggestion as well. Let me apply this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org