[jira] [Created] (YARN-11118) Containers not allocated although sufficient resources available

["Dirk Daems (Jira)" <ji...@apache.org>]

Dirk Daems created YARN-11118:
---------------------------------

             Summary: Containers not allocated although sufficient resources available
                 Key: YARN-11118
                 URL: https://issues.apache.org/jira/browse/YARN-11118
             Project: Hadoop YARN
          Issue Type: Bug
    Affects Versions: 3.1.1
            Reporter: Dirk Daems


We run Spark 2.3.2 on YARN 3.1.1 using the CapacityScheduler and Dominant Resource Calculator. Queue priorities are not being used.

We intermittently run into an issue where the PySpark driver is started but the executor containers can't be allocated although sufficient cluster resources are available. The YARN Docker container runtime is being used.

We already checked the following:
 * queue's user limit factor is not preventing containers to be allocated
 * max # applications threshold is not reached (both cluster and queue level)
 * max allocation vcores and memory not reached
 * max AM resource threshold is not reached (but not applicable as AM is already started)
 * queue's max capacity is not reached and cluster has sufficient resources available
 * cpu core and memory resources are not scattered: on several NodeManager nodes allocation of containers with requested cores and memory should be possible

Killing the job and restarting it with the exact same configuration will immediately start the driver; executor containers are allocated without a problem. This also seems to indicate that it's not a configuration mistake, but a scheduling bug.

Dumping YARN scheduler logs doesn't output information explaining the cause of this behavior.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org