(spark) branch branch-3.5 updated: [MINOR][CORE][TEST] Fix flaky DAGSchedulerSuite test for push-based shuffle

[ho...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.5 by this push:
     new 653bda4fb425 [MINOR][CORE][TEST] Fix flaky DAGSchedulerSuite test for push-based shuffle
653bda4fb425 is described below

commit 653bda4fb42509ec96f5c35c1b41b94df073063e
Author: Holden Karau <ho...@pigscanfly.ca>
AuthorDate: Wed Sep 23 22:22:21 2026 -0700

    [MINOR][CORE][TEST] Fix flaky DAGSchedulerSuite test for push-based shuffle
    
    ### What changes were proposed in this pull request?
    
    This is a backport of #55221 to `branch-3.5`.
    
    Add `sc.listenerBus.waitUntilEmpty()` before the `completedStage` assertion in the push-based shuffle test in `DAGSchedulerSuite`.
    
    Cherry-picked from commit c7c78e06384484b2323a813d1df8800705a98f6e on `master`. The cherry-pick was clean; no adaptation was needed, as the no-arg `LiveListenerBus.waitUntilEmpty()` overload already exists in `branch-3.5` and is used in ~20 other places in this same suite.
    
    ### Why are the changes needed?
    
    The test was intermittently failing due to a race condition. `SparkListenerStageCompleted` events are delivered asynchronously through the listener bus, so the `completedStage` assertion could execute before the event was processed, causing a spurious failure. Waiting for the listener bus to drain before asserting eliminates the race.
    
    The flake affects `branch-3.5` in the same way it affected `master`, so the fix is worth carrying over.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Test-only change.
    
    ### How was this patch tested?
    
    `build/sbt -Phive package` then `core/testOnly *DAGSchedulerSuite*` on `branch-3.5` under Java 17 (Temurin 17.0.16): 127 tests, all passed.
    
    The affected test — `SPARK-40082: recomputation of shuffle map stage with no pending partitions should finalize the stage. pushBasedShuffleEnabled = true` — was additionally run 10 more times in a row to check for residual flakiness; 10/10 green.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Sonnet 4.6 (Claude Code)
    
    Closes #59006 from holdenk/dagscheduler-flaky-fix-3.5.
    
    Lead-authored-by: Holden Karau <ho...@pigscanfly.ca>
    Co-authored-by: DB Tsai <db...@dbtsai.com>
    Signed-off-by: Holden Karau <ho...@snowflake.com>
---
 core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala | 1 +
 1 file changed, 1 insertion(+)

diff --git a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index 696e442bff6d..21104d4fa2a1 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -4967,6 +4967,7 @@ class DAGSchedulerSuite extends SparkFunSuite with TempLocalSparkContext with Ti
       // Map stage completes successfully,
       completeShuffleMapStageSuccessfully(0, 0, 3, Seq("hostA", "hostB"))
       taskIdCount += 2
+      sc.listenerBus.waitUntilEmpty()
       assert(completedStage === List(0))
 
       // Now submit the first reducer stage


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org