You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/11/02 14:07:27 UTC
[jira] [Assigned] (SPARK-11457) Yarn AM proxy filter configuration
should be reloaded when recovered from checkpoint
[ https://issues.apache.org/jira/browse/SPARK-11457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-11457:
------------------------------------
Assignee: (was: Apache Spark)
> Yarn AM proxy filter configuration should be reloaded when recovered from checkpoint
> ------------------------------------------------------------------------------------
>
> Key: SPARK-11457
> URL: https://issues.apache.org/jira/browse/SPARK-11457
> Project: Spark
> Issue Type: Bug
> Components: Streaming, YARN
> Affects Versions: 1.5.1
> Reporter: Saisai Shao
>
> Currently Yarn AM proxy filter configuration is recovered from checkpoint file when Spark Streaming application is restarted, which will lead to some unwanted behaviors:
> 1. Wrong RM address if RM is redeployed from failure.
> 2. Wrong proxyBase, since app id is updated, old app id for proxyBase is wrong.
> So instead of recovering from checkpoint file, these configurations should be reloaded each time when app started.
> This problem only exists in Yarn cluster mode, for Yarn client mode, these configurations will be updated with RPC message {{AddWebUIFilter}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org