[jira] [Created] (SPARK-9252) Spark client application must be restarted if the cluster (yarn) gets restarted

["Sarto Mihai (JIRA)" <ji...@apache.org>]

Sarto Mihai created SPARK-9252:
----------------------------------

             Summary: Spark client application must be restarted if the cluster (yarn) gets restarted
                 Key: SPARK-9252
                 URL: https://issues.apache.org/jira/browse/SPARK-9252
             Project: Spark
          Issue Type: Bug
          Components: Java API
    Affects Versions: 1.3.0
         Environment: Spark 1.3.0, Apache Hadoop 2.6
            Reporter: Sarto Mihai


We have a Java application that is building and sending successful RDDs. But if the cluster gets restarted, even we detect that from the application and rebuild the JavaSparkContext the execution will all fail until we restart the application too.
We are suspecting there is something static with the JavaSparkContext that does not get reinitialized - because we build new JavaSparkContext objects if we detect the oldSparkContext.env().isStopped().
If we also restart the 'client' application then the RDD executions will work just fine.
Therefore, we would like to not restart our application in case the Hadoop cluster get restarted and be able to make new JavaSparkContext in case the old Yarn application (Spark) was stopped.

Let me know should you need any more details.            



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org