[GitHub] [spark] LuciferYang commented on a diff in pull request #37487: [SPARK-40053][CORE][SQL][TESTS] Ignore EOL versions test and restrict test Python version for `HiveExternalCatalogVersionsSuite`

[GitBox <gi...@apache.org>]

LuciferYang commented on code in PR #37487:
URL: https://github.com/apache/spark/pull/37487#discussion_r945089983


##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala:
##########
@@ -249,11 +254,18 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
 }
 
 object PROCESS_TABLES extends QueryTest with SQLTestUtils {
+  val isPythonVersionAtLeast37 = TestUtils.isPythonVersionAtLeast37()
   val releaseMirror = sys.env.getOrElse("SPARK_RELEASE_MIRROR",
     "https://dist.apache.org/repos/dist/release")
   // Tests the latest version of every release line.
-  val testingVersions: Seq[String] = {
+  val testingVersions: Seq[String] = if (isPythonVersionAtLeast37) {
     import scala.io.Source
+    // SPARK-40053: This set needs to be redefined when there are version releases or EOL.
+    val testableVersions = if (SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)) {
+      Set("3.2, 3.3")
+    } else {
+      Set("3.1, 3.2, 3.3")

Review Comment:
   revert to
   ```
   versions.filter(v => !v.startsWith("2.4") && !v.startsWith("3.0"))
         .filter(v => !(v.startsWith("3.1") && SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
   ```
   
   Manual test:
   
   ```
   scala> val versions = Seq("2.4.8", "3.0.3", "3.1.2", "3.2.2", "3.3.0")
   val versions: Seq[String] = List(2.4.8, 3.0.3, 3.1.2, 3.2.2, 3.3.0)
   
   // SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17) is true
   scala> versions.filter(v => !v.startsWith("2.4") && !v.startsWith("3.0"))
        |   .filter(v => !(v.startsWith("3.1") && true))
   val res0: Seq[String] = List(3.2.2, 3.3.0)
   
   // SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17) is false
   scala> versions.filter(v => !v.startsWith("2.4") && !v.startsWith("3.0"))
        |       .filter(v => !(v.startsWith("3.1") && false))
   val res1: Seq[String] = List(3.1.2, 3.2.2, 3.3.0)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org