[jira] [Commented] (SPARK-14846) Driver process fails to terminate when graceful shutdown is used

["Shixiong Zhu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254628#comment-15254628 ] 

Shixiong Zhu commented on SPARK-14846:
--------------------------------------

`awaitTermination` doesn't need to wait at least one hour. `jobExecutor.shutdown()` is called before `jobExecutor.awaitTermination`. So when all threads in `jobExecutor` are done, awaitTermination will return.

> Driver process fails to terminate when graceful shutdown is used
> ----------------------------------------------------------------
>
>                 Key: SPARK-14846
>                 URL: https://issues.apache.org/jira/browse/SPARK-14846
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.6.1
>            Reporter: Mattias Aspholm
>
> During shutdown, the job scheduler in Streaming (JobScheduler.stop) spends some time waiting for all queued work to complete. If graceful shutdown is used, the time is 1 hour, for non-graceful shutdown it's 2 seconds.
> The wait is implemented using the ThreadPoolExecutor.awaitTermination method in java.util.concurrent. The problem is that instead of looping over the method for the desired period of time, the wait period is passed in as the timeout parameter to awaitTermination. 
> The result is that if the termination condition is false the first time, the method will sleep for the timeout period before trying again. In the case of graceful shutdown this means at least an hour's wait before the condition is checked again, even though all work is completed in just a few seconds. The driver process will continue to live during this time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org