You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/07/22 23:10:06 UTC
[jira] [Commented] (SPARK-9252) Spark client application must be
restarted if the cluster (yarn) gets restarted
[ https://issues.apache.org/jira/browse/SPARK-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637632#comment-14637632 ]
Sean Owen commented on SPARK-9252:
----------------------------------
I think this is more of a question for user@ rather than a JIRA.
The model for a Spark app is that the lifetime of the context is basically the lifetime of the driver JVM. If you're manually embedding Spark, yes you have to try to manage the context yourself. Although it should be possible to make a new context like that, I don't know that this is supported.
Try 1.4.1 instead. It's possible something was improved along the way to enabled this. If so it would be useful if you can provide any info about what the problem is, like what error you see, and therefore what is not being reinitialized.
> Spark client application must be restarted if the cluster (yarn) gets restarted
> -------------------------------------------------------------------------------
>
> Key: SPARK-9252
> URL: https://issues.apache.org/jira/browse/SPARK-9252
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 1.3.0
> Environment: Spark 1.3.0, Apache Hadoop 2.6
> Reporter: Sarto Mihai
>
> We have a Java application that is building and sending successful RDDs. But if the cluster gets restarted, even we detect that from the application and rebuild the JavaSparkContext the execution will all fail until we restart the application too.
> We are suspecting there is something static with the JavaSparkContext that does not get reinitialized - because we build new JavaSparkContext objects if we detect the oldSparkContext.env().isStopped().
> If we also restart the 'client' application then the RDD executions will work just fine.
> Therefore, we would like to not restart our application in case the Hadoop cluster get restarted and be able to make new JavaSparkContext in case the old Yarn application (Spark) was stopped.
> Let me know should you need any more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org