You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2025/11/23 16:02:39 UTC
(spark) branch branch-4.1 updated: [SPARK-54469][DSTREAM][TESTS] Fix `StreamingContextSuite.stop slow receiver gracefully` test to clean up `SprakContext`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 2f8dcb858984 [SPARK-54469][DSTREAM][TESTS] Fix `StreamingContextSuite.stop slow receiver gracefully` test to clean up `SprakContext`
2f8dcb858984 is described below
commit 2f8dcb858984263a168c3df632c84c96b3215e17
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Nov 23 08:02:20 2025 -0800
[SPARK-54469][DSTREAM][TESTS] Fix `StreamingContextSuite.stop slow receiver gracefully` test to clean up `SprakContext`
### What changes were proposed in this pull request?
This PR aims to fix `StreamingContextSuite.stop slow receiver gracefully` test case to clean up `SprakContext`.
### Why are the changes needed?
According to the log, `20000s` setting is propagated to the other test case and causes flaky failures.
- https://github.com/apache/spark/actions/runs/19608517338/job/56150848587
```
- SPARK-22955 graceful shutdown shouldn't lead to job generation error *** FAILED ***
The code passed to failAfter did not complete within 20 minutes. (StreamingContextSuite.scala:861)
```
`20000s` setting is originated from `StreamingContextSuite.stop slow receiver gracefully`. We should close the SparkContext correctly.
https://github.com/apache/spark/blob/997525cae9b00e47626ce09f00e779b279551ace/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala#L342-L344
### Does this PR introduce _any_ user-facing change?
No behavior change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #53182 from dongjoon-hyun/SPARK-54469.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit cda51211f8d475f63865713df26827e9639981ea)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../test/scala/org/apache/spark/streaming/StreamingContextSuite.scala | 2 ++
1 file changed, 2 insertions(+)
diff --git a/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala b/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
index 564713af88ae..539e74d9a255 100644
--- a/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
+++ b/streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
@@ -365,6 +365,8 @@ class StreamingContextSuite
assert(runningCount > 0)
assert(runningCount == totalNumRecords)
Thread.sleep(100)
+ sc.stop()
+ LocalStreamingContext.ensureNoActiveSparkContext()
}
test ("registering and de-registering of streamingSource") {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org