You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2025/06/06 16:37:25 UTC
(spark) branch branch-4.0 updated: [SPARK-52265][SQL][TEST] Fix regex leading to empty PROCESS_TABLES.testingVersions in HiveExternalCatalogVersionsSuite
This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new d1c11184894a [SPARK-52265][SQL][TEST] Fix regex leading to empty PROCESS_TABLES.testingVersions in HiveExternalCatalogVersionsSuite
d1c11184894a is described below
commit d1c11184894a1b4f5ba2447eea79ef21c682cd38
Author: Emilie Faracci <ef...@amazon.com>
AuthorDate: Sat Jun 7 00:34:40 2025 +0800
[SPARK-52265][SQL][TEST] Fix regex leading to empty PROCESS_TABLES.testingVersions in HiveExternalCatalogVersionsSuite
### What changes were proposed in this pull request?
Fix the version parsing logic in `HiveExternalCatalogVersionsSuite` to properly handle new artifact paths in https://dist.apache.org/repos/dist/release/spark/ so that "backward compatibility" test can be run.
This change creates a constant `val SparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r` for more precise version matching, and removes redundant `.filterNot(_.contains("preview"))` which is no longer needed.
### Why are the changes needed?
The suite is failing to execute the "backward compatibility" test due to parsing errors with testing versions. The current implementation fails to parse versions when encountering new paths like `spark-connect-swift-0.1.0/` and `spark-kubernetes-operator-0.1.0/` in https://dist.apache.org/repos/dist/release/spark/.
This leads to `PROCESS_TABLES.testingVersions` being empty, and in turn a logError: "Exception encountered when invoking run on a nested suite - Fail to get the latest Spark versions to test". As a result, the condition is not met to run the test.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Executed local build and test for `HiveExternalCatalogVersionsSuite`:
`build/mvn -pl sql/hive-Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite test-compile scalatest:test`
Verified that the reported error no longer appears, "backward compatibility" test runs successfully, and `PROCESS_TABLES.testingVersions` now correctly contains "3.5.5" when printed out, which was previously empty.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50989 from efaracci018/fix-testingVersions.
Lead-authored-by: Emilie Faracci <ef...@amazon.com>
Co-authored-by: efaracci018 <ef...@amazon.com>
Signed-off-by: yangjie01 <ya...@baidu.com>
(cherry picked from commit a380380428dbfcd03d7d76b876b33b8534caa4c8)
Signed-off-by: yangjie01 <ya...@baidu.com>
---
.../spark/sql/hive/HiveExternalCatalogVersionsSuite.scala | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
index 684e4c53e4ef..32f2c6d508d3 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
@@ -96,7 +96,12 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
mirrors.distinct :+ "https://archive.apache.org/dist" :+ PROCESS_TABLES.releaseMirror
logInfo(s"Trying to download Spark $version from $sites")
for (site <- sites) {
- val filename = s"spark-$version-bin-hadoop3-scala2.13.tgz"
+ val scalaVersion = version match {
+ case v if v.startsWith("3.") => "-scala2.13"
+ case v if v.startsWith("4.") => ""
+ case _ => fail(s"Spark version $version is unexpected")
+ }
+ val filename = s"spark-$version-bin-hadoop3$scalaVersion.tgz"
val url = s"$site/spark/spark-$version/$filename"
logInfo(s"Downloading Spark $version from $url")
try {
@@ -262,13 +267,13 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
val testingVersions: Seq[String] = if (isPythonVersionAvailable &&
SystemUtils.isJavaVersionAtMost(JavaVersion.JAVA_17)) {
import scala.io.Source
+ val sparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r
try Utils.tryWithResource(
Source.fromURL(s"$releaseMirror/spark")) { source =>
source.mkString
.split("\n")
- .filter(_.contains("""<a href="spark-"""))
- .filterNot(_.contains("preview"))
- .map("""<a href="spark-(\d.\d.\d)/">""".r.findFirstMatchIn(_).get.group(1))
+ .filter(sparkVersionPattern.unanchored.matches(_))
+ .map(sparkVersionPattern.findFirstMatchIn(_).get.group(1))
.filter(_ < org.apache.spark.SPARK_VERSION)
.filterNot(skipReleaseVersions.contains).toImmutableArraySeq
} catch {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org