[spark] branch master updated: [SPARK-39407][SQL][TESTS] Fix ParquetIOSuite to handle the case where DNS responses on `nonexistent`

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new e1ae76077a2 [SPARK-39407][SQL][TESTS] Fix ParquetIOSuite to handle the case where DNS responses on `nonexistent`
e1ae76077a2 is described below

commit e1ae76077a253b5e7221c062c087f84024366b66
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Jun 7 17:24:16 2022 -0700

    [SPARK-39407][SQL][TESTS] Fix ParquetIOSuite to handle the case where DNS responses on `nonexistent`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to update `ParquetIOSuite` to handle the case where DNS responses on `nonexistent`.
    
    ### Why are the changes needed?
    
    In some environment, DNS responses on `nonexists` hostname of `hdfs://nonexistent`.
    ```
    % ping nonexists
    PING nonexists (....): 56 data bytes
    64 bytes from ...: icmp_seq=0 ttl=56 time=8.718 ms
    ```
    
    This leads to a test case failure like the following.
    ```
    - SPARK-6330 regression test *** FAILED ***
      "java.lang.IllegalArgumentException: Pathname  from hdfs://nonexistent is not a valid DFS filename." did not contain "UnknownHostException" (ParquetIOSuite.scala:1184)
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a test case change.
    
    ### How was this patch tested?
    
    Pass the CIs and manually tests.
    
    Closes #36795 from dongjoon-hyun/SPARK-39407.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../spark/sql/execution/datasources/parquet/ParquetIOSuite.scala       | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
index 6560db4cb17..5a8f4563756 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
@@ -1181,7 +1181,8 @@ class ParquetIOSuite extends QueryTest with ParquetTest with SharedSparkSession
     val errorMessage = intercept[Throwable] {
       spark.read.parquet("hdfs://nonexistent")
     }.toString
-    assert(errorMessage.contains("UnknownHostException"))
+    assert(errorMessage.contains("UnknownHostException") ||
+        errorMessage.contains("is not a valid DFS filename"))
   }
 
   test("SPARK-7837 Do not close output writer twice when commitTask() fails") {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org