You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2025/01/22 15:27:49 UTC

(spark) branch branch-4.0 updated: [SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new 89ec42048ec4 [SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass
89ec42048ec4 is described below

commit 89ec42048ec4290b4a697c43735acec8de955133
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Wed Jan 22 17:27:23 2025 +0200

    [SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass
    
    ### What changes were proposed in this pull request?
    This pr adds Java version checks and logic adaptations to `scenario6` in the test case named "SPARK-35907: createDirectory" within the `UtilsSuite`, ensuring that it can pass the tests both Java 17.0.13 and Java 17.0.14 as well.
    
    ### Why are the changes needed?
    Java 17.0.14 includes the fix for JDK-8294193: [`Files.createDirectories throws FileAlreadyExistsException for a symbolic link whose target is an existing directory`](https://github.com/openjdk/jdk17u-dev/pull/2870). This behaves differently from previous expectations in the targeted test scenario, which would cause the "SPARK-35907: createDirectory" test to fail.
    
    ![image](https://github.com/user-attachments/assets/21d33f5c-d29a-4def-ba84-41475b231a63)
    
    However, since `setup-java` does not always install Java 17.0.14 (sometimes it installs 17.0.13 instead now), the current pr adapts the logic for versions after 17.0.14 and retains the original assertion behavior for versions 17.0.13 and earlier.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    - Pass GitHub Actions
    - manual check
    
    **Before**
    
    ```
    java -version
    openjdk version "17.0.14" 2025-01-21 LTS
    OpenJDK Runtime Environment Zulu17.56+15-CA (build 17.0.14+7-LTS)
    OpenJDK 64-Bit Server VM Zulu17.56+15-CA (build 17.0.14+7-LTS, mixed mode, sharing)
    ```
    
    or
    
    ```
    java -version
    openjdk version "17.0.14" 2025-01-21 LTS
    OpenJDK Runtime Environment Corretto-17.0.14.7.1 (build 17.0.14+7-LTS)
    OpenJDK 64-Bit Server VM Corretto-17.0.14.7.1 (build 17.0.14+7-LTS, mixed mode, sharing)
    ```
    
    "SPARK-35907: createDirectory" will test failed:
    
    ```
    [info] - SPARK-35907: createDirectory *** FAILED *** (43 milliseconds)
    [info]   Utils.createDirectory(scenario6) was true (UtilsSuite.scala:534)
    [info]   org.scalatest.exceptions.TestFailedException:
    [info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
    [info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
    [info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
    [info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
    [info]   at org.apache.spark.util.UtilsSuite.$anonfun$new$35(UtilsSuite.scala:534)
    [info]   at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
    [info]   at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
    [info]   at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
    [info]   at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
    [info]   at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69)
    [info]   at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155)
    [info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
    [info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
    [info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
    [info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
    [info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
    [info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
    [info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
    [info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69)
    [info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
    [info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
    [info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
    [info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
    [info]   at scala.collection.immutable.List.foreach(List.scala:334)
    [info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
    [info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
    [info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
    [info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
    [info]   at org.scalatest.Suite.run(Suite.scala:1114)
    [info]   at org.scalatest.Suite.run$(Suite.scala:1096)
    [info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
    [info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
    [info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
    [info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69)
    [info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
    [info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
    [info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
    [info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
    [info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
    [info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
    [info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
    [info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    [info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    [info]   at java.base/java.lang.Thread.run(Thread.java:840)
    ```
    
    **After**
    
    Both 17.0.13 and 17.0.14 will pass the test successfully.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #49599 from LuciferYang/SPARK-50946.
    
    Authored-by: yangjie01 <ya...@baidu.com>
    Signed-off-by: Max Gekk <ma...@gmail.com>
    (cherry picked from commit 54a59b7f3ceb575e478650ab8ead01922595ea17)
    Signed-off-by: Max Gekk <ma...@gmail.com>
---
 core/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
index baa748573d75..b026462c20c8 100644
--- a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
+++ b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
@@ -526,10 +526,15 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties {
 
     // The following 3 scenarios are only for the method: createDirectory(File)
     // 6. Symbolic link
+    // JAVA_RUNTIME_VERSION is like "17.0.14+7-LTS"
+    lazy val javaVersion = Runtime.Version.parse(SystemUtils.JAVA_RUNTIME_VERSION)
     val scenario6 = java.nio.file.Files.createSymbolicLink(new File(testDir, "scenario6")
       .toPath, scenario1.toPath).toFile
     if (Utils.isJavaVersionAtLeast21) {
       assert(Utils.createDirectory(scenario6))
+    } else if (javaVersion.feature() == 17 && javaVersion.update() >= 14) {
+      // SPARK-50946: Java 17.0.14 includes JDK-8294193, so scenario6 can succeed.
+      assert(Utils.createDirectory(scenario6))
     } else {
       assert(!Utils.createDirectory(scenario6))
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org