You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hean-Chhinling (via GitHub)" <gi...@apache.org> on 2026/08/12 08:03:50 UTC
[PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Hean-Chhinling opened a new pull request, #8674:
URL: https://github.com/apache/hadoop/pull/8674
### Description of PR
Utilise leaf-template capacity for displaying the capacity of auto created queue.
There are two different type of leaf-template capacity configurations:
1. In Weight mode -> Flexible Mode
```
<property>
<name>yarn.scheduler.capacity.root.default.auto-queue-creation-v2.leaf-template.capacity</name>
<value>3w</value>
</property>
```
2. In relative and absolute mode -> Legacy mode
```
<property>
<name>yarn.scheduler.capacity.root.default.leaf-queue-template.capacity</name>
<value>100</value>
</property>
```
Therefore, the PR handle both of this cases and taking the capacity to the auto created child queue.
### How was this patch tested?
Tested locally and works as expected. Please refer to the screenshot below:
<img width="2007" height="1341" alt="Screenshot 2026-08-12 at 10 02 26" src="https://github.com/user-attachments/assets/7a9b9549-0ee9-47c3-9bcf-2890df82ab67" />
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ X] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ X] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3864319921
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
+ } else if (configuredCapacity > 0) {
+ capacityConfig = String(configuredCapacity);
+ }
+ }
+
+ if (!maxCapacityDisplay.isStaged) {
+ const configuredMaxCapacity =
+ queueInfo.maxCapacity ?? defaultResourcePartition?.maxCapacity ?? 0;
+ const configuredMaxResource = defaultResourcePartition?.configuredMaxResource;
+
+ if (
+ configuredMaxResource &&
+ (configuredMaxResource.memory > 0 || configuredMaxResource.vCores > 0)
+ ) {
+ maxCapacityConfig = `[memory-mb=${configuredMaxResource.memory},vcores=${configuredMaxResource.vCores}]`;
Review Comment:
Thanks you, Beni! Yep we could expose this from backend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5268364258
> Thanks @Hean-Chhinling for fixing this bug!
>
> If i see correctly you checked in the
>
> > The PR includes the phrase "Contains content generated by "
> > where is the name of the AI tool used.
>
> field on the PR, but i dont see this in github. Do i miss something?
Thanks you for the review @K0K0V0K.
I missed that statement now I added it at the top of my PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3845787435
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
Review Comment:
The scheduler REST response already provides queueCapacityVectorInfo.configuredCapacityVector. Reconstructing the value from weight, configuredMinResource, and the computed capacity is lossy. For example, a valid mixed vector such as [memory-mb=16384,vcores=100%] has weight = -1 and may have configuredMinResource = 0/0, so this code displays the computed scalar capacity instead. It also drops custom resources (like GPU/FPGA/etc). Could we type and use configuredCapacityVector directly?
The general idea is the less custom logic on the UI the better, we should rely on the RM as much as possible, so that if something changes we don't need to constantly update the UI logic as well.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
Review Comment:
This seems a bit redundant, why do we need the isAutoCreatedLeafQueue check?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
+ } else if (configuredCapacity > 0) {
+ capacityConfig = String(configuredCapacity);
+ }
+ }
+
+ if (!maxCapacityDisplay.isStaged) {
+ const configuredMaxCapacity =
+ queueInfo.maxCapacity ?? defaultResourcePartition?.maxCapacity ?? 0;
+ const configuredMaxResource = defaultResourcePartition?.configuredMaxResource;
+
+ if (
+ configuredMaxResource &&
+ (configuredMaxResource.memory > 0 || configuredMaxResource.vCores > 0)
+ ) {
+ maxCapacityConfig = `[memory-mb=${configuredMaxResource.memory},vcores=${configuredMaxResource.vCores}]`;
Review Comment:
The same issue applies to maximum-capacity: non-legacy queue mode allows a mixed capacity vector, but configuredMaxResource cannot preserve percentage/weight components or custom resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5429603378
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 25s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 42s | | trunk passed |
| +1 :green_heart: | compile | 3m 26s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 3m 24s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 23s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 18m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 8s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 8s | | the patch passed |
| +1 :green_heart: | compile | 3m 23s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 18s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui has no data from spotbugs |
| +1 :green_heart: | shadedclient | 17m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 92m 12s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/10/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 1m 0s | | hadoop-yarn-capacity-scheduler-ui in the patch passed. |
| -1 :x: | asflicense | 0m 44s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/10/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 191m 53s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerConfigMutation |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreationAbsoluteMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigWeightModeDQC |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigWeightMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreation |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentage |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModePercentageAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModePercentageAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigAbsoluteMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDefaultLabel |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets compile javac javadoc mvninstall mvnsite unit spotbugs checkstyle |
| uname | Linux 5c578824041d 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 83ccf7f6b4f44e9f2fde4ba6893dcda2dc9ac043 |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/10/testReport/ |
| Max. process+thread count | 970 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/10/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5266844070
Thanks @Hean-Chhinling for fixing this bug!
If i see correctly you checked in the
> The PR includes the phrase "Contains content generated by "
> where is the name of the AI tool used.
field on the PR, but i dont see this in github. Do i miss something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3813056839
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/utils/templateUtils.ts:
##########
@@ -45,3 +49,85 @@ export function isTemplateQueuePath(queuePath: string): boolean {
queuePath.includes(marker),
);
}
+
+export type QueuePropertyLookup = (
+ queuePath: string,
+ property: string,
+) => {
+ value: string;
+ isStaged: boolean;
+};
+
+export type ResolvedQueueCapacityConfig = {
+ capacityConfig: string;
+ maxCapacityConfig: string;
+};
+
+const isAutoCreationEnabled = (
+ getQueuePropertyValue: QueuePropertyLookup,
+ queuePath: string,
+ property: string,
+): boolean => getQueuePropertyValue(queuePath, property).value.trim() === 'true';
+
+/**
+ * Resolves the template queue path for an auto-created child queue based on the
+ * parent's auto-creation settings:
+ * - flexible v2 (`auto-queue-creation-v2.enabled`) → leaf-template
+ * - legacy (`auto-create-child-queue.enabled`) → leaf-queue-template
Review Comment:
The issue is that we cannot solely rely on the leaf-template. AQCv2's template handling logic takes multiple values into account (template, wildcards, leaf-template), and this change assumes that nothing else matters. See in my comment below: https://github.com/apache/hadoop/pull/8674#issuecomment-5341896606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5409545713
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 26m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 38s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 42m 58s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 15m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 31s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/8/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 60m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/8/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux 4f2282741ef9 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 33237971fa49f7ba0a0f4e2790dfdd7b792e8acc |
| Max. process+thread count | 636 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/8/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5265272292
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 83m 9s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 33m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. |
| | | 121m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux 30ed9ccb92c6 5.15.0-185-generic #195-Ubuntu SMP Fri Jun 19 17:11:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9dc9ba79217431ec5bac8035ead91cdfa61750f2 |
| Max. process+thread count | 589 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/1/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3812784948
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/utils/templateUtils.ts:
##########
@@ -45,3 +49,85 @@ export function isTemplateQueuePath(queuePath: string): boolean {
queuePath.includes(marker),
);
}
+
+export type QueuePropertyLookup = (
+ queuePath: string,
+ property: string,
+) => {
+ value: string;
+ isStaged: boolean;
+};
+
+export type ResolvedQueueCapacityConfig = {
+ capacityConfig: string;
+ maxCapacityConfig: string;
+};
+
+const isAutoCreationEnabled = (
+ getQueuePropertyValue: QueuePropertyLookup,
+ queuePath: string,
+ property: string,
+): boolean => getQueuePropertyValue(queuePath, property).value.trim() === 'true';
+
+/**
+ * Resolves the template queue path for an auto-created child queue based on the
+ * parent's auto-creation settings:
+ * - flexible v2 (`auto-queue-creation-v2.enabled`) → leaf-template
+ * - legacy (`auto-create-child-queue.enabled`) → leaf-queue-template
Review Comment:
Thank you for the review @brumi1024!
I purposely not handle the other because in this PR, it is for fixing the bug that the auto leaf-queue capacity is not showing correctly. Therefore, this is to check only for the leaf templates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5346020823
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 48m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 15m 18s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 66m 49s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux c062e48da967 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5d6319823bc5e5aca67c68ed45821ed2d9e4149 |
| Max. process+thread count | 636 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/4/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3812784948
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/utils/templateUtils.ts:
##########
@@ -45,3 +49,85 @@ export function isTemplateQueuePath(queuePath: string): boolean {
queuePath.includes(marker),
);
}
+
+export type QueuePropertyLookup = (
+ queuePath: string,
+ property: string,
+) => {
+ value: string;
+ isStaged: boolean;
+};
+
+export type ResolvedQueueCapacityConfig = {
+ capacityConfig: string;
+ maxCapacityConfig: string;
+};
+
+const isAutoCreationEnabled = (
+ getQueuePropertyValue: QueuePropertyLookup,
+ queuePath: string,
+ property: string,
+): boolean => getQueuePropertyValue(queuePath, property).value.trim() === 'true';
+
+/**
+ * Resolves the template queue path for an auto-created child queue based on the
+ * parent's auto-creation settings:
+ * - flexible v2 (`auto-queue-creation-v2.enabled`) → leaf-template
+ * - legacy (`auto-create-child-queue.enabled`) → leaf-queue-template
Review Comment:
Thank you for the review @brumi1024!
I purposely not handle the others because in this PR, it is for fixing the bug that the auto leaf-queue capacity is not showing correctly. Therefore, this is to check only for the leaf templates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3851749652
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
Review Comment:
Thanks for actively reviewing the PR @brumi1024!
I did not use `configuredCapacityVector` intentionally because when I checked the /scheduler endpoint it shows as the following despite I configured the auto queue created capacity to (2048,2):
```
<queueCapacityVectorInfo>
<configuredCapacityVector>[memory-mb=0.0%,vcores=0.0%]</configuredCapacityVector>
</queueCapacityVectorInfo>
```
And at `configuredMinResource` it shows correctly, therefore I used it instead:
```
<configuredMinResource>
<memory>2048</memory>
<vCores>2</vCores>
</configuredMinResource>
```
Shouldn't the `configuredMinResource` is the right one here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3854822746
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
Review Comment:
Yes, this is flexible queue mode, but legacy AQC, I was referring to the flexible AQC.
What about something like this? https://github.com/brumi1024/hadoop/commit/7ae0528045cba1f7ed4ff08adc3b60577b8684d6
With this we wouldn't require parsing the config again on the UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5429770402
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 9s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 0s | | trunk passed |
| +1 :green_heart: | compile | 3m 35s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 3m 25s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 23s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 18m 11s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 3m 25s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 25s | | the patch passed |
| +1 :green_heart: | compile | 3m 24s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 18s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui has no data from spotbugs |
| +1 :green_heart: | shadedclient | 17m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 92m 46s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/11/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 1m 1s | | hadoop-yarn-capacity-scheduler-ui in the patch passed. |
| -1 :x: | asflicense | 0m 43s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/11/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 191m 55s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerConfigMutation |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreationAbsoluteMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigWeightModeDQC |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigWeightMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreation |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentage |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModePercentageAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModePercentageAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigAbsoluteMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDefaultLabel |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets compile javac javadoc mvninstall mvnsite unit spotbugs checkstyle |
| uname | Linux 2b6950416b63 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 60cf053940aee391f92396d81dbfb8220374adfe |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/11/testReport/ |
| Max. process+thread count | 957 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/11/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3851881141
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
Review Comment:
This was a safety for some reason when `creationMethod` is not present which is very unlikely, I will remove it in my next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3851806362
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
+ } else if (configuredCapacity > 0) {
+ capacityConfig = String(configuredCapacity);
+ }
+ }
+
+ if (!maxCapacityDisplay.isStaged) {
+ const configuredMaxCapacity =
+ queueInfo.maxCapacity ?? defaultResourcePartition?.maxCapacity ?? 0;
+ const configuredMaxResource = defaultResourcePartition?.configuredMaxResource;
+
+ if (
+ configuredMaxResource &&
+ (configuredMaxResource.memory > 0 || configuredMaxResource.vCores > 0)
+ ) {
+ maxCapacityConfig = `[memory-mb=${configuredMaxResource.memory},vcores=${configuredMaxResource.vCores}]`;
Review Comment:
Hmm..there is no entries that I could see for `maxCapacityVector` or something similar at the scheduler REST API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5411891681
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 26m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 15m 16s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 32s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/9/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 59m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux 586e97758fda 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62033e26fafa0834e251e516cb033d9d7ceaae70 |
| Max. process+thread count | 636 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/9/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3852336293
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
Review Comment:
For the legacy queue mode? Yes. For the non-legacy one? We should show what is provided. And this logic omits any custom resources we might have:
`capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;`
It was a while ago I touched the UI, but I think there should already be a logic that parses and shows the resources from a resource object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5346730910
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 18s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 70m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 27m 35s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. |
| | | 120m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux 221de03ca1cd 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e00cc5b2dc8591ad2dace9ba636832f5f294611b |
| Max. process+thread count | 609 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/5/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3864346010
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
Review Comment:
Thank you so much for the help, @brumi1024!
That is really nice approach to handle the logic at /store instead of /hooks. Also exposing the maxCapacityVector at the backend.
I have applied your patch, added a unit test then did a manual testing and all seems to be working as expected:
<img width="1949" height="1214" alt="Screenshot 2026-08-26 at 16 42 39" src="https://github.com/user-attachments/assets/4467c8bd-1682-4c1f-bceb-022c88ed974c" />
```
<queueCapacitiesByPartition>
<partitionName/>
<queueCapacityVectorInfo>
<configuredCapacityVector>[memory-mb=2048.0,vcores=2.0]</configuredCapacityVector>
<capacityVectorEntries>
<resourceName>memory-mb</resourceName>
<resourceValue>2048.0</resourceValue>
</capacityVectorEntries>
<capacityVectorEntries>
<resourceName>vcores</resourceName>
<resourceValue>2.0</resourceValue>
</capacityVectorEntries>
</queueCapacityVectorInfo>
<maximumQueueCapacityVectorInfo>
<configuredCapacityVector>[memory-mb=4096.0,vcores=4.0]</configuredCapacityVector>
<capacityVectorEntries>
<resourceName>memory-mb</resourceName>
<resourceValue>4096.0</resourceValue>
</capacityVectorEntries>
<capacityVectorEntries>
<resourceName>vcores</resourceName>
<resourceValue>4.0</resourceValue>
</capacityVectorEntries>
</maximumQueueCapacityVectorInfo>
</queueCapacitiesByPartition>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5398592649
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 45m 32s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 45m 50s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 15m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 27s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/7/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 64m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/7/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux 3c90841f7753 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 33237971fa49f7ba0a0f4e2790dfdd7b792e8acc |
| Max. process+thread count | 726 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/7/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5398593823
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 45m 57s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 46m 16s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 15m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | asflicense | 0m 27s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/6/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 64m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/6/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux d5ebbe921c4d 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 33237971fa49f7ba0a0f4e2790dfdd7b792e8acc |
| Max. process+thread count | 617 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/6/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5346048057
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 48m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 18m 33s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not generate ASF License warnings. |
| | | 70m 34s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux 3c904bb3e6a8 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5d6319823bc5e5aca67c68ed45821ed2d9e4149 |
| Max. process+thread count | 642 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/3/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3811969819
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/utils/templateUtils.ts:
##########
@@ -45,3 +49,85 @@ export function isTemplateQueuePath(queuePath: string): boolean {
queuePath.includes(marker),
);
}
+
+export type QueuePropertyLookup = (
+ queuePath: string,
+ property: string,
+) => {
+ value: string;
+ isStaged: boolean;
+};
+
+export type ResolvedQueueCapacityConfig = {
+ capacityConfig: string;
+ maxCapacityConfig: string;
+};
+
+const isAutoCreationEnabled = (
+ getQueuePropertyValue: QueuePropertyLookup,
+ queuePath: string,
+ property: string,
+): boolean => getQueuePropertyValue(queuePath, property).value.trim() === 'true';
+
+/**
+ * Resolves the template queue path for an auto-created child queue based on the
+ * parent's auto-creation settings:
+ * - flexible v2 (`auto-queue-creation-v2.enabled`) → leaf-template
+ * - legacy (`auto-create-child-queue.enabled`) → leaf-queue-template
Review Comment:
I agree with @pan3793's comment. This doesn't seem to handle parent-template, template and wildcard scenarios. See the docs about flexible AQC: https://github.com/apache/hadoop/blob/7f491908020e7dccc6fcdad3574e0fd0b6af2b2c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md?plain=1#L815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3851898947
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -252,6 +252,46 @@ function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible';
+
+ if (isAutoCreatedQueue) {
+ const resourcePartitions = queueInfo.capacities?.queueCapacitiesByPartition;
+ const defaultResourcePartition = resourcePartitions?.length
+ ? (resourcePartitions.find((entry) => !entry.partitionName) ?? resourcePartitions[0])
+ : undefined;
+
+ if (capacityDisplay.isStaged) {
+ // Pending local edit from Edit Capacity — not yet applied to the cluster.
+ capacityConfig = capacityDisplay.value;
+ } else {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else {
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+ if (configuredCapacity > 0) {
+ capacityConfig = String(configuredCapacity);
+ }
Review Comment:
I am resolving this thread as we are discussing the implementation of handling absolute mode memory in other thread.
Thank you for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3854810492
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
+ } else if (configuredCapacity > 0) {
+ capacityConfig = String(configuredCapacity);
+ }
+ }
+
+ if (!maxCapacityDisplay.isStaged) {
+ const configuredMaxCapacity =
+ queueInfo.maxCapacity ?? defaultResourcePartition?.maxCapacity ?? 0;
+ const configuredMaxResource = defaultResourcePartition?.configuredMaxResource;
+
+ if (
+ configuredMaxResource &&
+ (configuredMaxResource.memory > 0 || configuredMaxResource.vCores > 0)
+ ) {
+ maxCapacityConfig = `[memory-mb=${configuredMaxResource.memory},vcores=${configuredMaxResource.vCores}]`;
Review Comment:
If there is none we can expose it :) I generally would avoid trying to solve anything in the UI, as if YARN drifts the UI will need to be updated separately, which usually is forgotten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5345293536
Thank you so much @pan3793 and @brumi1024 for the review!
I have turned to utilise the existing queueInfo so that way we will not duplicate the parsing schedulerConfigs.
Please feel free to review again when you get time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5340132586
Scanned with an LLM and posted the selected feedback below
DISCLAIMER: I review each finding before posting, they look reasonable to me literally, but I'm not an expert in the frontend area, please re-evaluate each item carefully (you can reject the item with reasons if you think it's wrong).
Findings
1. Auto-created parents show the wrong template. getAutoCreatedQueueTemplatePath (templateUtils.ts:78) always picks FLEXIBLE_LEAF for every dynamicFlexible queue. But flexible auto-creation makes parents dynamic too -
CapacitySchedulerInfoHelper documents "both a parent and a leaf can be either static or dynamicFlexible", and parent capacity comes from auto-queue-creation-v2.parent-template (AutoCreatedQueueTemplate.parentOnlyProperties). So for an app
targeting root.default.team.user, the auto-created root.default.team card shows the leaf weight (3w) instead of the parent weight. Fix: branch on queueInfo.queueType and use FLEXIBLE_PARENT for parents.
2. Template overrides the queue's own staged value. resolveAutoCreatedQueueCapacityConfigs (templateUtils.ts:122-129) unconditionally replaces the queue value with the template value. The card's "Edit Capacity" is enabled for auto-created
queues (QueueCardContextMenu.tsx:126) and staging a change writes capacity on the queue path (capacityEditorSlice.ts:431); getQueuePropertyValue is staged-aware (queueDataSlice.ts:37-51). After editing 3w -> 5w, the card still shows 3w. Pre-
trunk behavior showed the staged value, so this is a regression of the change. Fix: use the template only when the queue's own value is empty/unset, and never when the own value is staged.
3. Nested/wildcard templates are missed. The helper reconstructs <parent>.auto-queue-creation-v2.leaf-template and looks it up in the config map. For depth >= 2 auto-created queues, the effective leaf-template lives at root.default.*.auto-
queue-creation-v2.leaf-template (wildcard inheritance, AutoCreatedQueueTemplate.setTemplateConfigEntries), so the lookup returns empty and the 0% display persists. The RM already resolves per-queue template properties and ships them in the
scheduler REST as autoQueueLeafTemplateProperties / autoQueueParentTemplateProperties (present in the UI's CapacitySchedulerInfo type, types/scheduler.ts:90-92, though not yet on QueueInfo). Reading those fields per queue would fix findings 1
and 3 at once and drop the path-guessing entirely.
4. Nit: formatting. templateUtils.ts:87 is 105 chars, over the repo printWidth: 100; prettier --check (format:check) would flag it. The two-blank-line import block also reads oddly.
5. Nit: test coverage. Tests are good and idiomatic, but they don't cover the parent-template case, staged-edit precedence, or nested/wildcard templates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5345928949
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 26s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | shadedclient | 18m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. |
| | | 71m 20s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets |
| uname | Linux f712d5d28e8a 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3596555f2752fe6ac636e6cf3802cd9947042b91 |
| Max. process+thread count | 613 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/2/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5475545800
Hello @brumi1024!
When you have time, may I ask for your review again on this PR?
This asflicense failure is not related to this PR [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/13/artifact/out/results-asflicense.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3843431462
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -252,6 +252,46 @@ function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible';
+
+ if (isAutoCreatedQueue) {
+ const resourcePartitions = queueInfo.capacities?.queueCapacitiesByPartition;
+ const defaultResourcePartition = resourcePartitions?.length
+ ? (resourcePartitions.find((entry) => !entry.partitionName) ?? resourcePartitions[0])
+ : undefined;
+
+ if (capacityDisplay.isStaged) {
+ // Pending local edit from Edit Capacity — not yet applied to the cluster.
+ capacityConfig = capacityDisplay.value;
+ } else {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else {
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+ if (configuredCapacity > 0) {
+ capacityConfig = String(configuredCapacity);
+ }
Review Comment:
Thanks for the update, looks better now. This still doesn't handle a few scenarios though:
- legacy AQC can be either percentage or absolute values
- flexible AQC in legacy queue mode can be weight only.
- flexible AQC in non-legacy queue mode can be anything, through the resource vectors. I.e [vcores=15%,memory=2w,gpu=4]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#discussion_r3853476138
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/features/queue-management/hooks/useQueueTreeData.ts:
##########
@@ -247,11 +247,59 @@ function getAutoCreationStatus(
}
function transformToCardData(queueInfo: QueueInfo, stagedChanges: StagedChange[]): QueueCardData {
- const getQueuePropertyValue = useSchedulerStore.getState().getQueuePropertyValue;
+ const { getQueuePropertyValue, getQueuePartitionCapacities } = useSchedulerStore.getState();
const capacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'capacity');
const maxCapacityDisplay = getQueuePropertyValue(queueInfo.queuePath, 'maximum-capacity');
+ let capacityConfig = capacityDisplay.value;
+ let maxCapacityConfig = maxCapacityDisplay.value;
+
+ const isAutoCreatedQueue =
+ queueInfo.creationMethod === 'dynamicLegacy' ||
+ queueInfo.creationMethod === 'dynamicFlexible' ||
+ (queueInfo as { isAutoCreatedLeafQueue?: boolean }).isAutoCreatedLeafQueue === true;
+
+ if (isAutoCreatedQueue) {
+ const defaultResourcePartition =
+ getQueuePartitionCapacities(queueInfo.queuePath, '') ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.find((entry) => !entry.partitionName) ??
+ queueInfo.capacities?.queueCapacitiesByPartition?.[0];
+ const configuredMinResource = defaultResourcePartition?.configuredMinResource;
+
+ if (!capacityDisplay.isStaged) {
+ const configuredWeight = queueInfo.weight ?? defaultResourcePartition?.weight;
+ const configuredCapacity =
+ queueInfo.capacity ?? defaultResourcePartition?.capacity ?? 0;
+
+ if (configuredWeight !== undefined && configuredWeight > 0) {
+ capacityConfig = `${configuredWeight}w`;
+ } else if (
+ configuredMinResource &&
+ (configuredMinResource.memory > 0 || configuredMinResource.vCores > 0)
+ ) {
+ capacityConfig = `[memory-mb=${configuredMinResource.memory},vcores=${configuredMinResource.vCores}]`;
Review Comment:
Thanks! I am able to test out successfully once I triggered the queue creation in mixed mode.
Now it shows correctly in the UI:
<img width="1614" height="898" alt="Screenshot 2026-08-25 at 15 27 52" src="https://github.com/user-attachments/assets/5201b6ce-7b95-462f-8bb5-46f3986d2670" />
Except for the MaxCapacity, I am not sure how would it be possible to handle mixedMode within /scheduler endpoint. Because the `maxCapacityVector` is not exist anywhere there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5328026420
Hello @brumi1024 and @slfan1989,
May I ask for you guys review here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5429731425
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 47s | | trunk passed |
| +1 :green_heart: | compile | 3m 29s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 27s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 16m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 17s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 3m 15s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 15s | | the patch passed |
| +1 :green_heart: | compile | 3m 27s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 22s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui has no data from spotbugs |
| +1 :green_heart: | shadedclient | 15m 34s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 91m 35s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/12/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m 58s | | hadoop-yarn-capacity-scheduler-ui in the patch passed. |
| -1 :x: | asflicense | 0m 45s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/12/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 187m 12s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerConfigMutation |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModePercentageAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentage |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreation |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigWeightMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreationAbsoluteMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigAbsoluteMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndWeight |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDefaultLabel |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModeAbsoluteAndPercentageAndWeightVector |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfigWeightModeDQC |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedModePercentageAndWeight |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets compile javac javadoc mvninstall mvnsite unit spotbugs checkstyle |
| uname | Linux 923e926959af 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 60cf053940aee391f92396d81dbfb8220374adfe |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/12/testReport/ |
| Max. process+thread count | 953 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/12/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Fix 0% capacity for auto created queue [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5444459001
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 44 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 291m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 134m 34s | | trunk passed |
| +1 :green_heart: | compile | 3m 24s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 3m 35s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 29s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 16m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 18s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 13s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 13s | | the patch passed |
| +1 :green_heart: | compile | 3m 31s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 3m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +0 :ok: | spotbugs | 0m 18s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui has no data from spotbugs |
| +1 :green_heart: | shadedclient | 16m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 91m 20s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 1m 0s | | hadoop-yarn-capacity-scheduler-ui in the patch passed. |
| -1 :x: | asflicense | 0m 44s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/13/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 584m 25s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8674 |
| Optional Tests | dupname asflicense shadedclient codespell detsecrets compile javac javadoc mvninstall mvnsite unit spotbugs checkstyle jsonlint xmllint |
| uname | Linux 20440c89b7a4 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8ec882336a687582e61cb83d7e4d54a475792cfb |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/13/testReport/ |
| Max. process+thread count | 968 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8674/13/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11981: Utilise leaf-template capacity for the auto created queue [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #8674:
URL: https://github.com/apache/hadoop/pull/8674#issuecomment-5341896606
Thanks for the fix, and the bug report is valid - the card really does show 0% for a queue that the RM is giving 3w. But I'd like to suggest a different approach, because the current one will keep missing cases.
The root cause is where the card gets its number. `transformToCardData` reads capacity through getQueuePropertyValue, which looks into the scheduler-conf store - that is the raw configuration a cluster admin wrote. An auto-created queue never has its own root.default.user1.capacity key there; the RM resolves the template internally (AutoCreatedQueueTemplate.setTemplateEntriesForChild) and applies it in memory. So any lookup on the config side will come up empty, and the UI falls back to '0'.
This PR works around that by rebuilding the template path in the UI (<parent>.auto-queue-creation-v2.leaf-template). The problem is that the RM's template resolution is more involved than that one key: there are template, leaf-template and parent-template with a precedence order, wildcard templates like root.default.*.auto-queue-creation-v2.leaf-template.capacity for deeper queues, dynamic parent queues that use parent-template, and a default of 1w when no template is configured at all (AbstractCSQueue.setupConfigurableCapacities). Reimplementing all of that in TypeScript means a second copy of the logic that will drift from the RM, the main thing I wanted to avoid in the config UI.
The good news is we don't have to: the /ws/v1/cluster/scheduler response already contains the RM-computed answer for every queue, and the UI already parses it into QueueInfo - look at weight, normalizedWeight and queueCapacityVectorInfo.configuredCapacityVector in types/queue.ts. They're just not used by the card yet.
So the suggestion: in transformToCardData, when creationMethod is dynamicFlexible or dynamicLegacy, take capacity / max capacity from those REST fields, and only let a staged config value (isStaged === true) override it, so edits still show before apply. That removes the need for the new template-path helper entirely, and it fixes wildcard, parent-template and the no-template default in one go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org