Re: [PR] [GLUTEN-11915][VL] Support RowBasedChecksum in ColumnarShuffleWriter (SPARK-51756) [gluten]

["philo-he (via GitHub)" <gi...@apache.org>]

philo-he commented on code in PR #12067:
URL: https://github.com/apache/gluten/pull/12067#discussion_r3346625358


##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenMapStatusEndToEndSuite.scala:
##########
@@ -28,5 +29,58 @@ class GlutenMapStatusEndToEndSuite extends MapStatusEndToEndSuite with GlutenTes
     _spark.sparkContext.conf
       .set(SQLConf.CLASSIC_SHUFFLE_DEPENDENCY_FILE_CLEANUP_ENABLED.key, "false")
     _spark.conf.set(SQLConf.CLASSIC_SHUFFLE_DEPENDENCY_FILE_CLEANUP_ENABLED.key, "false")
+
+    // Disable ANSI fallback to force Gluten's ColumnarShuffleWriter path.
+    _spark.conf.set("spark.gluten.sql.ansiFallback.enabled", "false")
+  }
+
+  import org.apache.spark.MapOutputTrackerMaster
+
+  private def getLatestShuffleChecksumValues(): Array[Long] = {
+    val tracker = _spark.sparkContext.env.mapOutputTracker
+      .asInstanceOf[MapOutputTrackerMaster]
+    val latestShuffleId = tracker.shuffleStatuses.keys.max
+    tracker.shuffleStatuses(latestShuffleId).mapStatuses.map(_.checksumValue)
+  }
+
+  test("Gluten row-based checksum is deterministic") {

Review Comment:
   Test suites like GlutenMapStatusEndToEndSuite extend Spark's own test classes to run Spark tests with Gluten enabled. For the newly added tests here, if they aren't variants of existing tests in the parent Spark test class, it may be better to move them to a dedicated test suite. Could you do a refactor?



##########
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala:
##########
@@ -1027,7 +1027,7 @@ class VeloxTestSettings extends BackendTestSettings {
     .exclude("SPARK-49386: test SortMergeJoin (with spill by size threshold)")
   enableSuite[GlutenMathFunctionsSuite]
   // TODO: fix on Spark-4.1 see https://github.com/apache/spark/pull/50230

Review Comment:
   Should we remove this comment?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org