(spark) branch branch-4.0 updated: [SPARK-52944][CORE][TESTS][FOLLOWUP] Avoid hard-coding the checksum algorithm name

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yangjie01 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new 1352e31ad245 [SPARK-52944][CORE][TESTS][FOLLOWUP] Avoid hard-coding the checksum algorithm name
1352e31ad245 is described below

commit 1352e31ad245b1efe89841afd43d8b94e2ee5d54
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Sat Jul 26 21:56:28 2025 -0700

    [SPARK-52944][CORE][TESTS][FOLLOWUP] Avoid hard-coding the checksum algorithm name
    
    ### What changes were proposed in this pull request?
    This PR retrieves the shuffle checksum algorithm name via `conf.get(SHUFFLE_CHECKSUM_ALGORITHM)` to eliminate hard-coding.
    
    ### Why are the changes needed?
    Address https://github.com/apache/spark/pull/51654#discussion_r2231775664
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    - Pass Github Actions
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #51675 from LuciferYang/SPARK-52944-FOLLOWUP.
    
    Authored-by: yangjie01 <ya...@baidu.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit afd595a57f1d7980cd58137c643ab73f7f6771a2)
    Signed-off-by: yangjie01 <ya...@baidu.com>
---
 core/src/test/scala/org/apache/spark/SortShuffleSuite.scala | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/core/src/test/scala/org/apache/spark/SortShuffleSuite.scala b/core/src/test/scala/org/apache/spark/SortShuffleSuite.scala
index b57fda538ff4..41dc67fb0ef5 100644
--- a/core/src/test/scala/org/apache/spark/SortShuffleSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SortShuffleSuite.scala
@@ -26,7 +26,7 @@ import org.apache.commons.io.filefilter.TrueFileFilter
 import org.scalatest.BeforeAndAfterAll
 import org.scalatest.matchers.should.Matchers._
 
-import org.apache.spark.internal.config.SHUFFLE_MANAGER
+import org.apache.spark.internal.config.{SHUFFLE_CHECKSUM_ALGORITHM, SHUFFLE_MANAGER}
 import org.apache.spark.rdd.ShuffledRDD
 import org.apache.spark.serializer.{JavaSerializer, KryoSerializer}
 import org.apache.spark.shuffle.sort.SortShuffleManager
@@ -70,7 +70,8 @@ class SortShuffleSuite extends ShuffleSuite with BeforeAndAfterAll {
     // Ensure that the shuffle actually created files that will need to be cleaned up
     val filesCreatedByShuffle = getAllFiles -- filesBeforeShuffle
     filesCreatedByShuffle.map(_.getName) should be(
-      Set("shuffle_0_0_0.data", "shuffle_0_0_0.checksum.ADLER32", "shuffle_0_0_0.index"))
+      Set("shuffle_0_0_0.data", s"shuffle_0_0_0.checksum.${conf.get(SHUFFLE_CHECKSUM_ALGORITHM)}",
+        "shuffle_0_0_0.index"))
     // Check that the cleanup actually removes the files
     sc.env.blockManager.master.removeShuffle(0, blocking = true)
     for (file <- filesCreatedByShuffle) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org