You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/05/24 04:10:44 UTC
[spark] branch master updated: [SPARK-43767][SQL][TESTS] Fix bug in AvroSuite for 'reading from invalid path throws exception'
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a72d1aa4910 [SPARK-43767][SQL][TESTS] Fix bug in AvroSuite for 'reading from invalid path throws exception'
a72d1aa4910 is described below
commit a72d1aa4910c67c3b8b4dd78997818ef6a601875
Author: panbingkun <pb...@gmail.com>
AuthorDate: Tue May 23 21:10:29 2023 -0700
[SPARK-43767][SQL][TESTS] Fix bug in AvroSuite for 'reading from invalid path throws exception'
### What changes were proposed in this pull request?
The pr aims to fix bug in AvroSuite for 'reading from invalid path throws exception'.
### Why are the changes needed?
- As discussed and analyzed in [41271](https://github.com/apache/spark/pull/41271#issuecomment-1560355918)
- There is a problem with this UT. Its original intention was to test if there is no file with .avro extensions in the directory, and the read should fail. However, this UT triggered the error as FileUtils.touch instead of spark.read.format("avro").load(dir.toString).The root cause for the failure of this case is that the parent directory was not created. When FileUtils.touch is called in version 1.11.0, it just throws java.io.FileNotFoundException, which covers the error.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
Closes #41289 from panbingkun/SPARK-43767.
Authored-by: panbingkun <pb...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index d19a11b4546..a09cbf53ab9 100644
--- a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -1350,7 +1350,7 @@ abstract class AvroSuite
}
intercept[FileNotFoundException] {
- withTempPath { dir =>
+ withTempDir { dir =>
FileUtils.touch(new File(dir, "test"))
withSQLConf(AvroFileFormat.IgnoreFilesWithoutExtensionProperty -> "true") {
spark.read.format("avro").load(dir.toString)
@@ -1359,7 +1359,7 @@ abstract class AvroSuite
}
intercept[FileNotFoundException] {
- withTempPath { dir =>
+ withTempDir { dir =>
FileUtils.touch(new File(dir, "test"))
spark
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org