You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/09/07 02:07:01 UTC
[jira] [Assigned] (SPARK-21933) Spark Streaming request more
executors than excepted without DynamicAllocation
[ https://issues.apache.org/jira/browse/SPARK-21933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-21933:
------------------------------------
Assignee: Apache Spark
> Spark Streaming request more executors than excepted without DynamicAllocation
> ------------------------------------------------------------------------------
>
> Key: SPARK-21933
> URL: https://issues.apache.org/jira/browse/SPARK-21933
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.2.0
> Reporter: Congxian Qiu
> Assignee: Apache Spark
>
> When Spark Streaming application runs on Yarn without DynamicAllocation, If some nodemanager becomes lost, then the containers on the lost nodemanager will be reported to all the applicationmaster, application master will allocate new containers.
> But after application master allocate new containers, the lost nodemanager be available, after this, resource manager restarted, after resource manager has been restarted, the node manager will report all the containers running on it before to resource manager because of Yarn's HA, then application manager will receive a duplicated completed container message, so the spark streaming application will request more resource than it requires.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org