You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by zs...@apache.org on 2016/04/23 00:42:53 UTC
spark git commit: [SPARK-14701][STREAMING] First stop the event loop,
then stop the checkpoint writer in JobGenerator
Repository: spark
Updated Branches:
refs/heads/master 3647120a5 -> fde1340c7
[SPARK-14701][STREAMING] First stop the event loop, then stop the checkpoint writer in JobGenerator
Currently if we call `streamingContext.stop` (e.g. in a `StreamingListener.onBatchCompleted` callback) when a batch is about to complete, a `rejectedException` may get thrown from `checkPointWriter.executor`, since the `eventLoop` will try to process `DoCheckpoint` events even after the `checkPointWriter.executor` was stopped.
Please see [SPARK-14701](https://issues.apache.org/jira/browse/SPARK-14701) for details and stack traces.
## What changes were proposed in this pull request?
Reversed the stopping order of `event loop` and `checkpoint writer`.
## How was this patch tested?
Existing test suits.
(no dedicated test suits were added because the change is simple to reason about)
Author: Liwei Lin <lw...@gmail.com>
Closes #12489 from lw-lin/spark-14701.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fde1340c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fde1340c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fde1340c
Branch: refs/heads/master
Commit: fde1340c768e18e9628e5f0eeb2f283c74c294fa
Parents: 3647120
Author: Liwei Lin <lw...@gmail.com>
Authored: Fri Apr 22 15:42:47 2016 -0700
Committer: Shixiong Zhu <sh...@databricks.com>
Committed: Fri Apr 22 15:42:47 2016 -0700
----------------------------------------------------------------------
.../org/apache/spark/streaming/scheduler/JobGenerator.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/fde1340c/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
----------------------------------------------------------------------
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
index 307ff1f..8f9421f 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
@@ -154,9 +154,9 @@ class JobGenerator(jobScheduler: JobScheduler) extends Logging {
graph.stop()
}
- // Stop the event loop and checkpoint writer
- if (shouldCheckpoint) checkpointWriter.stop()
+ // First stop the event loop, then stop the checkpoint writer; see SPARK-14701
eventLoop.stop()
+ if (shouldCheckpoint) checkpointWriter.stop()
logInfo("Stopped JobGenerator")
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org