[jira] [Created] (YARN-12020) AQC v2 template user weights not applied to dynamic queues

["Benjamin Teke (Jira)" <ji...@apache.org>]

Benjamin Teke created YARN-12020:
------------------------------------

             Summary: AQC v2 template user weights not applied to dynamic queues
                 Key: YARN-12020
                 URL: https://issues.apache.org/jira/browse/YARN-12020
             Project: Hadoop YARN
          Issue Type: Sub-task
          Components: capacity scheduler
    Affects Versions: 3.5.0
            Reporter: Benjamin Teke
            Assignee: Benjamin Teke


The v2 auto queue creation templates (auto-queue-creation-v2.template, leaf-template, parent-template) are documented to accept any property from the Queue Properties section, and yarn.scheduler.capacity.<queue-path>.user-settings.<user-name>.weight is one of them. A template user weight is written into the dynamic leaf's configuration, but the leaf does not use it.


Repro: root.a with auto-queue-creation-v2.enabled=true and
{noformat}
yarn.scheduler.capacity.root.a.auto-queue-creation-v2.template.user-limit-factor=3
yarn.scheduler.capacity.root.a.auto-queue-creation-v2.template.user-settings.u1.weight=0.5
{noformat}
then create the dynamic leaf root.a.auto1. getUserLimitFactor() of the leaf returns 3, but getUserWeights().getByUser("u1") returns the default 1.0 instead of 0.5, so the user limit of u1 is computed with the wrong weight.


Cause: AutoCreatedQueueTemplate writes the template entries into the shared queue configuration while the dynamic queue is set up. Most queue properties are then read with ordinary Configuration getters and see them, but CapacitySchedulerConfiguration#getAllUserWeightsForQueue reads through the cached prefix index of the configuration (ConfigurationProperties, ConfigSnapshot after YARN-12013), which is built before any template entry is written, so it never contains the template user weights. The index must not simply be rebuilt during queue setup: the explicitly-set check of the templates relies on it not containing the dynamic queue defaults written before the templates (for example user-limit-factor -1).
Expected: template user weights apply to dynamic queues like the other template properties, from both template and leaf-template entries.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org