[spark] branch branch-3.1 updated: [SPARK-34508][SQL][TEST] Skip HiveExternalCatalogVersionsSuite if network is down

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
     new 6e60c2d  [SPARK-34508][SQL][TEST] Skip HiveExternalCatalogVersionsSuite if network is down
6e60c2d is described below

commit 6e60c2ddd23ac5274b752a6b27e9a7c353d5daee
Author: Wenchen Fan <we...@databricks.com>
AuthorDate: Tue Feb 23 13:35:29 2021 -0800

    [SPARK-34508][SQL][TEST] Skip HiveExternalCatalogVersionsSuite if network is down
    
    ### What changes were proposed in this pull request?
    
    It's possible that the network is down when running Spark tests, and it's annoying to see `HiveExternalCatalogVersionsSuite` keep failing.
    
    This PR proposes to skip this test suite if we can't get the latest Spark version from the Apache website.
    
    ### Why are the changes needed?
    
    Make the Spark tests more robust.
    
    ### Does this PR introduce _any_ user-facing change?
    
    no
    
    ### How was this patch tested?
    
    N/A
    
    Closes #31627 from cloud-fan/test.
    
    Authored-by: Wenchen Fan <we...@databricks.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
    (cherry picked from commit 0d5d248bdc4cdc71627162a3d20c42ad19f24ef4)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala  | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
index cc278ea..5442062 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
@@ -191,7 +191,7 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
     // scalastyle:on line.size.limit
 
     if (PROCESS_TABLES.testingVersions.isEmpty) {
-      fail("Fail to get the lates Spark versions to test.")
+      logError("Fail to get the latest Spark versions to test.")
     }
 
     PROCESS_TABLES.testingVersions.zipWithIndex.foreach { case (version, index) =>
@@ -229,7 +229,7 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
       "--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
       "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath}",
       unusedJar.toString)
-    runSparkSubmit(args)
+    if (PROCESS_TABLES.testingVersions.nonEmpty) runSparkSubmit(args)
   }
 }
 
@@ -248,8 +248,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
         .map("""<a href="spark-(\d.\d.\d)/">""".r.findFirstMatchIn(_).get.group(1))
         .filter(_ < org.apache.spark.SPARK_VERSION)
     } catch {
-      // do not throw exception during object initialization.
-      case NonFatal(_) => Seq("3.0.1", "2.4.7") // A temporary fallback to use a specific version
+      // Do not throw exception during object initialization.
+      case NonFatal(_) => Nil
     }
     versions
       .filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org