You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/01/27 07:18:26 UTC
[spark] branch branch-3.1 updated: [SPARK-34231][AVRO][TEST] Make
proper use of resource file within AvroSuite test case
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 374aeef [SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite test case
374aeef is described below
commit 374aeef0ef89d5ad42d0e590175be53fd4ac220f
Author: Erik Krogen <xk...@apache.org>
AuthorDate: Wed Jan 27 16:16:33 2021 +0900
[SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite test case
Change `AvroSuite."Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES"` to use `episodesAvro`, which is loaded as a resource using the classloader, instead of trying to read `episodes.avro` directly from a relative file path.
This is the proper way to read resource files, and currently this test will fail when called from my IntelliJ IDE, though it will succeed when called from Maven/sbt, presumably due to different working directory handling.
No, unit test only.
Previous failure from IntelliJ:
```
Source 'src/test/resources/episodes.avro' does not exist
java.io.FileNotFoundException: Source 'src/test/resources/episodes.avro' does not exist
at org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1405)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1072)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1040)
at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34(AvroSuite.scala:397)
at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34$adapted(AvroSuite.scala:388)
```
Now it succeeds.
Closes #31332 from xkrogen/xkrogen-SPARK-34231-avrosuite-testfix.
Authored-by: Erik Krogen <xk...@apache.org>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit b2c104bd87361e5d85f7c227c60419af16b718f2)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index 1ac0fb2..b192b3d 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -392,11 +392,11 @@ abstract class AvroSuite
}.getMessage
assert(message.contains("No Avro files found."))
- val srcFile = new File("src/test/resources/episodes.avro")
- val destFile = new File(dir, "episodes.avro")
- FileUtils.copyFile(srcFile, destFile)
+ Files.copy(
+ Paths.get(new URL(episodesAvro).toURI),
+ Paths.get(dir.getCanonicalPath, "episodes.avro"))
- val result = spark.read.format("avro").load(srcFile.getAbsolutePath).collect()
+ val result = spark.read.format("avro").load(episodesAvro).collect()
checkAnswer(spark.read.format("avro").load(dir.getAbsolutePath), result)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org