[jira] [Created] (SPARK-15652) Missing org.apache.spark.launcher.SparkAppHandle.Listener notification if SparkSubmit JVM shutsdown

["Subroto Sanyal (JIRA)" <ji...@apache.org>]

Subroto Sanyal created SPARK-15652:
--------------------------------------

             Summary: Missing org.apache.spark.launcher.SparkAppHandle.Listener notification if SparkSubmit JVM shutsdown
                 Key: SPARK-15652
                 URL: https://issues.apache.org/jira/browse/SPARK-15652
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.6.0
            Reporter: Subroto Sanyal
            Priority: Critical


h6. Problem
In case SparkSubmit JVM goes down even before sending the job complete notification; the _org.apache.spark.launcher.SparkAppHandle.Listener_ will not receive any notification which may lead to the client using SparkLauncher hang indefinitely.
h6. Root Cause
No proper exception handling at org.apache.spark.launcher.LauncherConnection#run when an EOFException is encountered while reading over Socket Stream. Mostly EOFException will be thrown at the suggested point if the SparkSubmit JVM is shutdown. 
Probably, it was assumed that SparkSubmit JVM can shut down only with normal healthy completion but, there could be scenarios where this is not the case:
# OS kill the SparkSubmit process using OOM Killer.
# Exception while SparkSubmit submits the job, even before it starts monitoring the application. This can happen if SparkLauncher is not configured properly. There might be no exception handling in org.apache.spark.deploy.yarn.Client#submitApplication(), which may lead to any exception/throwable at this point lead to shutting down of JVM without proper finalisation
h6. Possible Solutions
# In case of EOFException or any other exception notify the listeners that job has failed
# Better exception handling on the SparkSubmit JVM side (though this may not resolve the problem completely)





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org