[jira] [Created] (SPARK-3456) YarnAllocator can lose container requests to RM

["Thomas Graves (JIRA)" <ji...@apache.org>]

Thomas Graves created SPARK-3456:
------------------------------------

             Summary: YarnAllocator can lose container requests to RM
                 Key: SPARK-3456
                 URL: https://issues.apache.org/jira/browse/SPARK-3456
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 1.2.0
            Reporter: Thomas Graves
            Priority: Critical


I haven't actually tested this yet, but I believe that spark on yarn can lose container requests to the RM.  The reason is that we ask for the total number upfront (say x) but then we don't ask for anymore unless some are missing and if we do then we could erase the original request.

For example

- ask for 3 containers
- 1 is allocated
- ask for 0 containers since asked for 3 originally (2 left)
- the 1 allocated dies
- We now ask for 1 since its missing, this will override whatever is on the yarn side (in this case 2).

Then we lose the 2 more we need.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org