You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2019/04/05 23:38:17 UTC
[spark] branch branch-2.4 updated: [SPARK-27391][SS] Don't
initialize a lazy val in ContinuousExecution job.
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new baadfc8 [SPARK-27391][SS] Don't initialize a lazy val in ContinuousExecution job.
baadfc8 is described below
commit baadfc81a752b07cd09a0ebb2c62a8ac45f32251
Author: Jose Torres <to...@gmail.com>
AuthorDate: Fri Apr 5 12:56:36 2019 -0700
[SPARK-27391][SS] Don't initialize a lazy val in ContinuousExecution job.
## What changes were proposed in this pull request?
Fix a potential deadlock in ContinuousExecution by not initializing the toRDD lazy val.
Closes #24301 from jose-torres/deadlock.
Authored-by: Jose Torres <to...@gmail.com>
Signed-off-by: Jose Torres <to...@gmail.com>
(cherry picked from commit 4a5768b2a2adf87e3df278655918f72558f0b3b9)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../sql/execution/streaming/continuous/ContinuousExecution.scala | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala
index 3037c01..0b332c2 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala
@@ -258,12 +258,8 @@ class ContinuousExecution(
epochUpdateThread.start()
reportTimeTaken("runContinuous") {
- SQLExecution.withNewExecutionId(
- sparkSessionForQuery, lastExecution) {
- // Materialize `executedPlan` so that accessing it when `toRdd` is running doesn't need to
- // wait for a lock
- lastExecution.executedPlan
- lastExecution.toRdd
+ SQLExecution.withNewExecutionId(sparkSessionForQuery, lastExecution) {
+ lastExecution.executedPlan.execute()
}
}
} catch {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org