[jira] [Created] (SPARK-11296) FutureAction.onComplete consumes a thread while waiting for job

["Richard W. Eggert II (JIRA)" <ji...@apache.org>]

Richard W. Eggert II created SPARK-11296:
--------------------------------------------

             Summary: FutureAction.onComplete consumes a thread while waiting for job
                 Key: SPARK-11296
                 URL: https://issues.apache.org/jira/browse/SPARK-11296
             Project: Spark
          Issue Type: Bug
          Components: Scheduler, Spark Core
    Affects Versions: 1.5.1, 1.5.0, 1.4.1, 1.4.0
            Reporter: Richard W. Eggert II


The implementations of the onComplete callback (and, by extension, the callbacks based on it) of FutureAction (SimpleFutureAction and ComplexFutureAction) each consume a thread that blocks waiting for the underlying job(s) to complete. This is unnecessary and is likely to be very surprising to anyone accustomed to the usual behavior of Scala Futures (i.e., that callbacks only occupy threads AFTER the underlying task has completed),  and it runs the risk of causing thread starvation for any processes utilizing the same thread pool (especially if scala.concurrent.ExecutionContext.global is used). The implementations can and should be reworked to operate in a non-blocking manner, by modifying the underlying JobWaiter class to expose a non-blocking API, so that the onComplete (as well as onSuccess and onFailure) callbacks only utilize threads while they are actually executing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org