[jira] [Resolved] (SPARK-3456) YarnAllocator can lose container requests to RM

["Thomas Graves (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves resolved SPARK-3456.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.2.0
         Assignee: Thomas Graves

> YarnAllocator can lose container requests to RM
> -----------------------------------------------
>
>                 Key: SPARK-3456
>                 URL: https://issues.apache.org/jira/browse/SPARK-3456
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.2.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Critical
>             Fix For: 1.2.0
>
>
> I haven't actually tested this yet, but I believe that spark on yarn can lose container requests to the RM.  The reason is that we ask for the total number upfront (say x) but then we don't ask for anymore unless some are missing and if we do then we could erase the original request.
> For example
> - ask for 3 containers
> - 1 is allocated
> - ask for 0 containers since asked for 3 originally (2 left)
> - the 1 allocated dies
> - We now ask for 1 since its missing, this will override whatever is on the yarn side (in this case 2).
> Then we lose the 2 more we need.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org