You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2025/11/23 16:02:39 UTC

(spark) branch branch-4.1 updated: [SPARK-54469][DSTREAM][TESTS] Fix `StreamingContextSuite.stop slow receiver gracefully` test to clean up `SprakContext`

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new 2f8dcb858984 [SPARK-54469][DSTREAM][TESTS] Fix `StreamingContextSuite.stop slow receiver gracefully` test to clean up `SprakContext`
2f8dcb858984 is described below

commit 2f8dcb858984263a168c3df632c84c96b3215e17
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Nov 23 08:02:20 2025 -0800

    [SPARK-54469][DSTREAM][TESTS] Fix `StreamingContextSuite.stop slow receiver gracefully` test to clean up `SprakContext`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `StreamingContextSuite.stop slow receiver gracefully` test case to clean up `SprakContext`.
    
    ### Why are the changes needed?
    
    According to the log, `20000s` setting is propagated to the other test case and causes flaky failures.
    - https://github.com/apache/spark/actions/runs/19608517338/job/56150848587
    ```
    - SPARK-22955 graceful shutdown shouldn't lead to job generation error *** FAILED ***
      The code passed to failAfter did not complete within 20 minutes. (StreamingContextSuite.scala:861)
    ```
    
    `20000s` setting is originated from `StreamingContextSuite.stop slow receiver gracefully`. We should close the SparkContext correctly.
    
    https://github.com/apache/spark/blob/997525cae9b00e47626ce09f00e779b279551ace/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala#L342-L344
    
    ### Does this PR introduce _any_ user-facing change?
    
    No behavior change.
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #53182 from dongjoon-hyun/SPARK-54469.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit cda51211f8d475f63865713df26827e9639981ea)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../test/scala/org/apache/spark/streaming/StreamingContextSuite.scala   | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala b/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
index 564713af88ae..539e74d9a255 100644
--- a/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
+++ b/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
@@ -365,6 +365,8 @@ class StreamingContextSuite
     assert(runningCount > 0)
     assert(runningCount == totalNumRecords)
     Thread.sleep(100)
+    sc.stop()
+    LocalStreamingContext.ensureNoActiveSparkContext()
   }
 
   test ("registering and de-registering of streamingSource") {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org