You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Congxian Qiu (JIRA)" <ji...@apache.org> on 2017/09/13 09:04:00 UTC
[jira] [Updated] (SPARK-21933) Spark Streaming request more
executors than excepted without DynamicAllocation
[ https://issues.apache.org/jira/browse/SPARK-21933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Congxian Qiu updated SPARK-21933:
---------------------------------
Description:
Spark Streaming applications run on Yarn without DynamicAllocation, Yarn enable HA.
When Spark Streaming applications run on Yarn without DynamicAllocation, If some nodemanager becomes lost, then the containers on the lost nodemanager will be reported to all the applicationmasters, application master will allocate new containers.
But after application master allocate new containers, the lost nodemanager become available, then, resource manager restarted, after resource manager has been restarted, the node manager will report the status of all the containers on it before to resource manager because of Yarn's HA, then application manager will receive a duplicated completed container message, and request more executors, so the spark streaming application will request more resource than it requires.
was:
When Spark Streaming application runs on Yarn without DynamicAllocation, If some nodemanager becomes lost, then the containers on the lost nodemanager will be reported to all the applicationmaster, application master will allocate new containers.
But after application master allocate new containers, the lost nodemanager be available, after this, resource manager restarted, after resource manager has been restarted, the node manager will report all the containers running on it before to resource manager because of Yarn's HA, then application manager will receive a duplicated completed container message, so the spark streaming application will request more resource than it requires.
> Spark Streaming request more executors than excepted without DynamicAllocation
> ------------------------------------------------------------------------------
>
> Key: SPARK-21933
> URL: https://issues.apache.org/jira/browse/SPARK-21933
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.2.0
> Reporter: Congxian Qiu
>
> Spark Streaming applications run on Yarn without DynamicAllocation, Yarn enable HA.
> When Spark Streaming applications run on Yarn without DynamicAllocation, If some nodemanager becomes lost, then the containers on the lost nodemanager will be reported to all the applicationmasters, application master will allocate new containers.
> But after application master allocate new containers, the lost nodemanager become available, then, resource manager restarted, after resource manager has been restarted, the node manager will report the status of all the containers on it before to resource manager because of Yarn's HA, then application manager will receive a duplicated completed container message, and request more executors, so the spark streaming application will request more resource than it requires.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org