You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/08/14 12:39:54 UTC
[jira] [Resolved] (SPARK-9396) Spark yarn allocator does not call
"removeContainerRequest" for allocated Container requests, resulting in
bloated ask[] toYarn RM.
[ https://issues.apache.org/jira/browse/SPARK-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-9396.
------------------------------
Resolution: Won't Fix
This is fixed in 1.3+ by other changes; unless a decision is taken to make another 1.2.x release I think this is a WontFix.
> Spark yarn allocator does not call "removeContainerRequest" for allocated Container requests, resulting in bloated ask[] toYarn RM.
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-9396
> URL: https://issues.apache.org/jira/browse/SPARK-9396
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.1, 1.2.2
> Environment: Spark-1.2.1 on hadoop-yarn-2.4.0 cluster. All servers in cluster running Linux version 2.6.32.
> Reporter: prakhar jauhari
> Priority: Minor
>
> Note : Attached logs contain logs that i added (spark yarn allocator side and Yarn client side) for debugging purpose.
> !!!!! My spark job is configured for 2 executors, on killing 1 executor the ask is of 3 !!!!!!!
> On killing a executor - resource request logs :
> *************Killed container: ask for 3 containers, instead for 1***********
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: Will allocate 1 executor containers, each with 2432 MB memory including 384 MB overhead
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: numExecutors: 1
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: host preferences is empty
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:2432, vCores:4>
> 15/07/15 10:49:01 INFO impl.AMRMClientImpl: prakhar : AMRMClientImpl : allocate: this.ask = [{Priority: 1, Capability: <memory:2432, vCores:4>, # Containers: 3, Location: *, Relax Locality: true}]
> 15/07/15 10:49:01 INFO impl.AMRMClientImpl: prakhar : AMRMClientImpl : allocate: allocateRequest = ask { priority{ priority: 1 } resource_name: "*" capability { memory: 2432 virtual_cores: 4 } num_containers: 3 relax_locality: true } blacklist_request { } response_id: 354 progress: 0.1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org