You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "brumi1024 (via GitHub)" <gi...@apache.org> on 2026/10/07 10:42:47 UTC
[PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
brumi1024 opened a new pull request, #8799:
URL: https://github.com/apache/hadoop/pull/8799
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Test-only groundwork for the configuration resolver work: focused tests in the existing Capacity Scheduler test classes for configuration checks, parsing, inheritance, auto-created queue templates, placement rules and refresh behavior that trunk did not cover or only covered by exception type, assertions on the exact rejection messages where tests only checked the exception type, and a load/refresh/validate benchmark that is skipped unless -DRunCapacitySchedulerConfigBenchmark=true is set.
For more details see: [YARN-12010](https://issues.apache.org/jira/browse/YARN-12010)
### How was this patch tested?
Unit test.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#issuecomment-6096773523
Thanks @slfan1989 and @joseluisll for the comments, addressed them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#issuecomment-6109057804
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 13m 25s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 30 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 16s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 30m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 57s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 551 unchanged - 2 fixed = 552 total (was 553) |
| +1 :green_heart: | mvnsite | 1m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 29m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 110m 34s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 246m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8799 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3587955b5f96 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c1f5fc5ff017dc71a62312f2f9d3add43977192a |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/4/testReport/ |
| Max. process+thread count | 938 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/4/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 merged PR #8799:
URL: https://github.com/apache/hadoop/pull/8799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#issuecomment-6092644988
@K0K0V0K @p-szucs If you have some time, I’d appreciate it if you could also take a look at this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#issuecomment-6040658984
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 13m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 30 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 53s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 551 unchanged - 2 fixed = 552 total (was 553) |
| +1 :green_heart: | mvnsite | 1m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 111m 44s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 253m 2s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8799 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2664a9592002 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 32f362f9db4cb87cae8f4f4ecb0321cd1ec7aef8 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/1/testReport/ |
| Max. process+thread count | 925 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#issuecomment-6120152862
Thanks @slfan1989 @joseluisll for the reviews, merging to trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "joseluisll (via GitHub)" <gi...@apache.org>.
joseluisll commented on code in PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#discussion_r4236878304
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerConfigBenchmark.java:
##########
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.resourcemanager.MockRM;
+import org.apache.hadoop.yarn.server.resourcemanager.NodeAttributeTestUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
+import org.junit.jupiter.api.Test;
+import org.slf4j.event.Level;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Capacity Scheduler configuration benchmark: scheduler load, queue refresh
+ * and configuration validation on generated trees of
+ * {@link CSConfigBenchmarkGenerator}.
+ *
+ * <p>It only runs with {@code -DRunCapacitySchedulerConfigBenchmark=true}.
+ * Each size should run in its own JVM, and each fork is a separate Maven
+ * invocation, for example three forks of every size:</p>
+ * <pre>
+ * RM=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
+ * for fork in 1 2 3; do for size in 10 100 1000 5000; do
+ * ./mvnw -o test -pl $RM -Dtest=TestCapacitySchedulerConfigBenchmark \
+ * -DRunCapacitySchedulerConfigBenchmark=true -Dcs.bench.sizes=$size \
+ * -Dcs.bench.fork=$fork -Dcs.bench.out=/tmp/cs-bench.txt
+ * done; done
+ * </pre>
+ * <p>Properties:</p>
+ * <ul>
+ * <li>{@code cs.bench.sizes}: requested queue counts, default
+ * {@code 10,100,1000,5000};</li>
+ * <li>{@code cs.bench.warmups} (default 2) and {@code cs.bench.iterations}
+ * (default 5) per operation; the median of the iterations is reported;</li>
+ * <li>{@code cs.bench.ops}: subset of {@code scheduler-load},
+ * {@code refresh} and {@code validate}, default all;</li>
+ * <li>{@code cs.bench.validators}: implementations measured by the
+ * {@code validate} operation, default {@code legacy}
+ * ({@link CapacitySchedulerConfigValidator#validateCSConfiguration}, the
+ * core of {@code POST /scheduler-conf/validate}); any other entry is the
+ * class name of a {@link ProposalValidator} with a no-argument constructor,
+ * which is the hook for {@code validate/v2};</li>
+ * <li>{@code cs.bench.capacityMode}: a
+ * {@link CSConfigBenchmarkGenerator.CapacityMode}, default
+ * {@code PCT_WEIGHT_MIXED};</li>
+ * <li>{@code cs.bench.fork}: label printed with the results;</li>
+ * <li>{@code cs.bench.out}: file the {@code BENCH} result lines are
+ * appended to (they are always printed to standard output).</li>
+ * </ul>
+ * <p>Operations: {@code scheduler-load} initializes a fresh
+ * {@link CapacityScheduler} from the configuration, {@code refresh}
+ * reinitializes the running scheduler with the unchanged configuration (what
+ * {@code AdminService.refreshQueues} does after loading the file), and
+ * {@code validate} validates a proposal that moves one percent of capacity
Review Comment:
Nit: in `ALL_ABSOLUTE` mode `createMutatedCopy` falls back to changing `maximum-applications` rather than moving capacity, so the validate proposal is not a one percent capacity move there. Worth mentioning in the javadoc.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSConfigBenchmarkGenerator.java:
##########
@@ -0,0 +1,543 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+
+import static org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableSet.of;
+
+/**
+ * Generates synthetic but structurally realistic capacity-scheduler
+ * configuration content for a requested queue count.
+ * It feeds {@link TestCapacitySchedulerConfigBenchmark} (scheduler load,
+ * refresh and validation timings) and {@link TestCSConfigBenchmarkGenerator}.
+ *
+ * Tree shape (mixed depth 3 and 4):
+ * <pre>
+ * root
+ * p0 (pct children, node label, one legacy managed parent leaf)
+ * p1 (weight children, AQC v2 template on sub-parent s0)
+ * p2 (pct children, unlabeled)
+ * p3 (weight children)
+ * ...
+ * each pI: 2 direct leaves + up to 5 sub-parents with up to 9 leaves each
+ * </pre>
+ *
+ * Capacity modes:
+ * - PCT_WEIGHT_MIXED (default): root children are percentage; even-indexed
+ * subtrees use percentage children, odd-indexed use weight children; the last
+ * sub-parent of a weight subtree flips its children back to percentage
+ * (legal per the legacy-mode parent/child mode table in AbstractParentQueue).
+ * - ALL_ABSOLUTE: every queue uses absolute [memory=..,vcores=..] resources,
+ * with parent minimums equal to the sum of their children. This is a separate
+ * mode because in legacy queue mode absolute resources cannot be mixed with
+ * percentage/weight anywhere in one hierarchy (absolute children require an
+ * absolute parent, and siblings must be mode-uniform).
+ *
+ * Extras sprinkled for realism: accessible-node-labels + per-label capacities
+ * on every fourth subtree (labels "blue"/"ssd"), legacy managed-parent
+ * (auto-create-child-queue) leaf templates, auto-queue-creation-v2 templates,
+ * legacy queue-mapping placement rules, ACLs, user-limit-factor,
+ * maximum-capacity and ordering policies on a subset of queues.
+ */
+public final class CSConfigBenchmarkGenerator {
+
+ public enum CapacityMode {
+ PCT_WEIGHT_MIXED,
+ ALL_ABSOLUTE
+ }
+
+ /** Everything the harness needs to know about a generated config. */
+ public static final class GeneratedConfig {
+ private final CapacitySchedulerConfiguration conf;
+ private final int queueCount;
+ private final List<String> leafPaths;
+ private final String mutationLeafA;
+ private final String mutationLeafB;
+ private final List<String> labels;
+
+ private GeneratedConfig(CapacitySchedulerConfiguration conf, int queueCount,
+ List<String> leafPaths, String mutationLeafA, String mutationLeafB,
+ List<String> labels) {
+ this.conf = conf;
+ this.queueCount = queueCount;
+ this.leafPaths = leafPaths;
+ this.mutationLeafA = mutationLeafA;
+ this.mutationLeafB = mutationLeafB;
+ this.labels = labels;
+ }
+
+ public CapacitySchedulerConfiguration getConf() {
+ return conf;
+ }
+
+ /** Number of generated queues, root excluded. */
+ public int getQueueCount() {
+ return queueCount;
+ }
+
+ public List<String> getLeafPaths() {
+ return Collections.unmodifiableList(leafPaths);
+ }
+
+ /** Two sibling leaves whose capacities can be swapped for a valid change. */
+ public String getMutationLeafA() {
+ return mutationLeafA;
+ }
+
+ public String getMutationLeafB() {
+ return mutationLeafB;
+ }
+
+ public List<String> getLabels() {
+ return Collections.unmodifiableList(labels);
+ }
+ }
+
+ private static final int DIRECT_LEAVES = 2;
+ private static final int SUB_PARENTS = 5;
+ private static final int LEAVES_PER_SUB_PARENT = 9;
+ private static final int MAX_MANAGED_PARENTS = 4;
+ private static final int MAX_AQC_V2_PARENTS = 4;
+ private static final String LABEL_BLUE = "blue";
+ private static final String LABEL_SSD = "ssd";
+ private static final String CS_PREFIX = CapacitySchedulerConfiguration.PREFIX;
+
+ private CSConfigBenchmarkGenerator() {
+ }
+
+ private static final class Node {
+ private final String name;
+ private final String path;
+ private final List<Node> children = new ArrayList<>();
+ private boolean managedParent;
+
+ Node(String parentPath, String name) {
+ this.name = name;
+ this.path = parentPath + "." + name;
+ }
+
+ boolean isLeaf() {
+ return children.isEmpty() && !managedParent;
+ }
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount) {
+ return generate(targetQueueCount, CapacityMode.PCT_WEIGHT_MIXED);
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount, CapacityMode mode) {
+ List<Node> tops = buildForest(targetQueueCount);
+ CapacitySchedulerConfiguration conf =
+ new CapacitySchedulerConfiguration(new Configuration(false), false);
+
+ markManagedParents(tops);
+ QueuePath rootPath = new QueuePath(CapacitySchedulerConfiguration.ROOT);
+ conf.setQueues(rootPath, names(tops));
+
+ if (mode == CapacityMode.ALL_ABSOLUTE) {
+ emitAbsolute(conf, rootPath, tops);
+ } else {
+ emitMixed(conf, tops);
+ }
+
+ List<String> leafPaths = new ArrayList<>();
+ int count = 0;
+ for (Node top : tops) {
+ count += collect(top, leafPaths);
+ }
+ String[] mutationPair = pickMutationSiblings(tops);
+ setPlacementRules(conf, leafPaths);
+
+ List<String> labels = new ArrayList<>();
+ if (mode == CapacityMode.PCT_WEIGHT_MIXED) {
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null && !labels.contains(label)) {
+ labels.add(label);
+ }
+ }
+ }
+ return new GeneratedConfig(conf, count, leafPaths, mutationPair[0],
+ mutationPair[1], labels);
+ }
+
+ /**
+ * Returns a copy of {@code base} with a valid capacity change applied: one
+ * percent is moved from mutation leaf B to mutation leaf A (and likewise for
+ * their per-label capacities), so per-parent sums stay at 100.
+ */
+ public static Configuration createMutatedCopy(Configuration base,
+ GeneratedConfig gen) {
+ Configuration copy = new Configuration(base);
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), null);
+ for (String label : gen.getLabels()) {
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), label);
+ }
+ return copy;
+ }
+
+ private static void moveOnePercent(Configuration conf, String leafA,
+ String leafB, String label) {
+ if (leafA == null || leafB == null) {
+ return;
+ }
+ String suffix = (label == null) ? ".capacity"
+ : ".accessible-node-labels." + label + ".capacity";
+ String keyA = CS_PREFIX + leafA + suffix;
+ String keyB = CS_PREFIX + leafB + suffix;
+ String a = conf.get(keyA);
+ String b = conf.get(keyB);
+ if (a == null || b == null) {
+ return;
+ }
+ try {
+ conf.set(keyA, String.valueOf(Float.parseFloat(a) + 1.0f));
+ conf.set(keyB, String.valueOf(Float.parseFloat(b) - 1.0f));
+ } catch (NumberFormatException e) {
+ // Absolute/weight profiles mutate a real queue limit, never a no-op.
+ conf.set(CS_PREFIX + leafA + ".maximum-applications", "2345");
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Tree construction
+ // ---------------------------------------------------------------------
+
+ private static List<Node> buildForest(int targetQueueCount) {
+ List<Node> tops = new ArrayList<>();
+ int fullSubtree = 1 + DIRECT_LEAVES + SUB_PARENTS * (1 + LEAVES_PER_SUB_PARENT);
+ int remaining = targetQueueCount;
+ int i = 0;
+ while (remaining > 0) {
+ int budget = Math.min(fullSubtree, remaining);
+ Node top = new Node(CapacitySchedulerConfiguration.ROOT, "p" + i);
+ budget--;
+ int directLeaves = Math.min(DIRECT_LEAVES, budget);
+ for (int l = 0; l < directLeaves; l++) {
+ top.children.add(new Node(top.path, "l" + l));
+ }
+ budget -= directLeaves;
+ int s = 0;
+ while (budget > 0) {
+ Node sub = new Node(top.path, "s" + s);
+ top.children.add(sub);
+ budget--;
+ int subLeaves = Math.min(LEAVES_PER_SUB_PARENT, budget);
+ for (int l = 0; l < subLeaves; l++) {
+ sub.children.add(new Node(sub.path, "l" + l));
+ }
+ budget -= subLeaves;
+ s++;
+ }
+ remaining -= subtreeSize(top);
+ tops.add(top);
+ i++;
+ }
+ return tops;
+ }
+
+ private static int subtreeSize(Node node) {
+ int size = 1;
+ for (Node child : node.children) {
+ size += subtreeSize(child);
+ }
+ return size;
+ }
+
+ /**
+ * Converts the last direct leaf of every other percentage subtree into a
+ * legacy managed parent (auto-create-child-queue.enabled), capped globally.
+ */
+ private static void markManagedParents(List<Node> tops) {
+ int marked = 0;
+ for (int i = 0; i < tops.size() && marked < MAX_MANAGED_PARENTS; i += 4) {
+ Node top = tops.get(i);
+ for (int c = top.children.size() - 1; c >= 0; c--) {
+ Node child = top.children.get(c);
+ if (child.isLeaf()) {
+ child.managedParent = true;
+ marked++;
+ break;
+ }
+ }
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Mixed percentage/weight emission
+ // ---------------------------------------------------------------------
+
+ private static void emitMixed(CapacitySchedulerConfiguration conf, List<Node> tops) {
+ float[] topCaps = splitPercentages(tops.size());
+ Map<String, List<Integer>> labelMembers = new LinkedHashMap<>();
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null) {
+ labelMembers.computeIfAbsent(label, k -> new ArrayList<>()).add(i);
+ }
+ }
+ // Per-label percentage shares among the labeled root children; the sums per
+ // label are 100 among label-accessible siblings, as legacy mode requires.
+ Map<Integer, Float> topLabelCap = new LinkedHashMap<>();
+ for (List<Integer> members : labelMembers.values()) {
+ float[] shares = splitPercentages(members.size());
+ for (int m = 0; m < members.size(); m++) {
+ topLabelCap.put(members.get(m), shares[m]);
+ }
+ }
+
+ int aqcV2Marked = 0;
+ for (int i = 0; i < tops.size(); i++) {
+ Node top = tops.get(i);
+ QueuePath topPath = new QueuePath(top.path);
+ conf.setCapacity(topPath, topCaps[i]);
+ conf.setMaximumCapacity(topPath, 100f);
+ String label = labelFor(i);
+ if (label != null) {
+ conf.setAccessibleNodeLabels(topPath, of(label));
+ conf.setCapacityByLabel(topPath, label, topLabelCap.get(i));
+ }
+ boolean weightMode = (i % 2 == 1);
+ boolean markAqcV2 = weightMode && (i % 6 == 1) && aqcV2Marked < MAX_AQC_V2_PARENTS;
+ if (markAqcV2) {
+ aqcV2Marked++;
+ }
+ emitChildren(conf, top, weightMode, label, true, markAqcV2);
+ }
+ }
+
+ /**
+ * Emits capacities (and per-label capacities) for the children of
+ * {@code parent}, then recurses one level into sub-parents.
+ */
+ private static void emitChildren(CapacitySchedulerConfiguration conf, Node parent,
+ boolean weightMode, String label, boolean topLevel, boolean markAqcV2) {
+ if (parent.children.isEmpty()) {
+ return;
+ }
+ conf.setQueues(new QueuePath(parent.path), names(parent.children));
+ float[] caps = splitPercentages(parent.children.size());
+ for (int c = 0; c < parent.children.size(); c++) {
+ Node child = parent.children.get(c);
+ QueuePath childPath = new QueuePath(child.path);
+ // The last sub-parent of a top-level weight subtree flips its children
+ // back to percentage mode; mode must stay uniform among siblings only.
+ boolean childrenWeightMode = weightMode
+ && !(topLevel && c == parent.children.size() - 1 && !child.children.isEmpty());
+ if (weightMode) {
+ conf.setNonLabeledQueueWeight(childPath, (c % 4) + 1f);
+ } else {
+ conf.setCapacity(childPath, caps[c]);
+ conf.setMaximumCapacity(childPath, 100f);
+ }
+ if (label != null) {
+ conf.setAccessibleNodeLabels(childPath, of(label));
+ conf.setCapacityByLabel(childPath, label, caps[c]);
+ }
+ decorate(conf, child, c);
+ if (child.managedParent) {
+ emitManagedParent(conf, childPath, label);
+ } else if (!child.children.isEmpty()) {
+ if (markAqcV2 && c == DIRECT_LEAVES) {
+ emitAqcV2Template(conf, childPath);
+ }
+ emitChildren(conf, child, childrenWeightMode, label, false, false);
+ }
+ }
+ }
+
+ private static void emitManagedParent(CapacitySchedulerConfiguration conf,
+ QueuePath path, String label) {
+ conf.setAutoCreateChildQueueEnabled(path, true);
+ conf.setAutoCreatedLeafQueueConfigCapacity(path, 25f);
+ conf.setAutoCreatedLeafQueueConfigMaxCapacity(path, 100f);
+ if (label != null) {
+ conf.setAutoCreatedLeafQueueTemplateCapacityByLabel(path, label, 25f);
+ }
+ }
+
+ private static void emitAqcV2Template(CapacitySchedulerConfiguration conf,
+ QueuePath path) {
+ conf.setAutoQueueCreationV2Enabled(path, true);
+ conf.set(CS_PREFIX + path.getFullPath()
+ + ".auto-queue-creation-v2.template.capacity", "2w");
+ conf.set(CS_PREFIX + path.getFullPath()
+ + ".auto-queue-creation-v2.leaf-template.maximum-applications", "100");
+ }
+
+ /** Realistic per-queue extras on a deterministic subset of queues. */
+ private static void decorate(CapacitySchedulerConfiguration conf, Node node,
+ int childIndex) {
+ QueuePath path = new QueuePath(node.path);
+ if (childIndex % 3 == 0) {
+ conf.setUserLimitFactor(path, 2.0f);
+ }
+ if (childIndex % 5 == 1) {
+ conf.setAcl(path, QueueACL.SUBMIT_APPLICATIONS, "benchuser benchgroup");
+ conf.setAcl(path, QueueACL.ADMINISTER_QUEUE, "benchadmin");
+ }
+ if (node.isLeaf() && childIndex % 4 == 2) {
+ conf.setOrderingPolicy(path, CapacitySchedulerConfiguration.FAIR_APP_ORDERING_POLICY);
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Absolute-mode emission
+ // ---------------------------------------------------------------------
+
+ private static void emitAbsolute(CapacitySchedulerConfiguration conf,
+ QueuePath rootPath, List<Node> tops) {
+ // Root keeps its fixed 100 percent capacity; setting an absolute capacity
+ // on root is rejected by CapacitySchedulerConfiguration.
+ for (Node top : tops) {
+ emitAbsoluteSubtree(conf, top);
+ }
+ }
+
+ /** Returns {memoryMb, vcores} configured for the subtree minimum. */
+ private static long[] emitAbsoluteSubtree(CapacitySchedulerConfiguration conf,
+ Node node) {
+ QueuePath path = new QueuePath(node.path);
+ if (node.children.isEmpty()) {
+ conf.setCapacity(path, "[memory=1024,vcores=1]");
+ conf.set(CS_PREFIX + node.path + ".maximum-capacity",
+ "[memory=2048,vcores=2]");
+ if (node.managedParent) {
+ conf.setAutoCreateChildQueueEnabled(path, true);
+ }
+ return new long[]{1024L, 1L};
+ }
+ conf.setQueues(path, names(node.children));
+ long mem = 0;
+ long vcores = 0;
+ for (Node child : node.children) {
+ long[] used = emitAbsoluteSubtree(conf, child);
+ mem += used[0];
+ vcores += used[1];
+ }
+ conf.setCapacity(path, "[memory=" + mem + ",vcores=" + vcores + "]");
+ conf.set(CS_PREFIX + node.path + ".maximum-capacity",
+ "[memory=" + (2 * mem) + ",vcores=" + (2 * vcores) + "]");
+ return new long[]{mem, vcores};
+ }
+
+ // ---------------------------------------------------------------------
+ // Placement rules and helpers
+ // ---------------------------------------------------------------------
+
+ private static void setPlacementRules(CapacitySchedulerConfiguration conf,
+ List<String> leafPaths) {
+ if (leafPaths.isEmpty()) {
+ return;
+ }
+ List<String> mappings = new ArrayList<>();
+ mappings.add("u:benchuser1:" + leafPaths.get(0));
+ mappings.add("u:benchuser2:" + leafPaths.get(leafPaths.size() / 2));
+ mappings.add("g:benchgroup:" + leafPaths.get(leafPaths.size() - 1));
+ if (leafPaths.size() > 3) {
+ mappings.add("u:%user:" + leafPaths.get(1));
+ }
+ conf.set(CapacitySchedulerConfiguration.QUEUE_MAPPING,
+ String.join(",", mappings));
+ }
+
+ private static int collect(Node node, List<String> leafPaths) {
+ int count = 1;
+ if (node.isLeaf()) {
+ leafPaths.add(node.path);
+ }
+ for (Node child : node.children) {
+ count += collect(child, leafPaths);
+ }
+ return count;
+ }
+
+ /**
+ * Picks two plain sibling leaves whose capacity keys can be swapped to
+ * produce a valid changed configuration. Prefers an unlabeled percentage
+ * subtree; falls back to any sub-parent, then to direct leaves.
+ */
+ private static String[] pickMutationSiblings(List<Node> tops) {
+ for (int pass = 0; pass < 2; pass++) {
+ for (int i = 0; i < tops.size(); i += 2) {
+ if (pass == 0 && labelFor(i) != null) {
+ continue;
+ }
+ Node top = tops.get(i);
+ for (Node child : top.children) {
+ List<Node> leaves = new ArrayList<>();
+ for (Node leaf : child.children) {
+ if (leaf.isLeaf()) {
+ leaves.add(leaf);
+ }
+ }
+ if (leaves.size() >= 2) {
+ return new String[]{leaves.get(0).path, leaves.get(1).path};
+ }
+ }
+ List<Node> direct = new ArrayList<>();
+ for (Node child : top.children) {
+ if (child.isLeaf()) {
+ direct.add(child);
+ }
+ }
+ if (direct.size() >= 2) {
+ return new String[]{direct.get(0).path, direct.get(1).path};
+ }
+ }
+ }
+ return new String[]{null, null};
+ }
+
+ private static String labelFor(int topIndex) {
+ if (topIndex % 4 != 0) {
+ return null;
+ }
+ return (topIndex % 8 == 0) ? LABEL_BLUE : LABEL_SSD;
+ }
+
+ private static String[] names(List<Node> nodes) {
+ String[] names = new String[nodes.size()];
+ for (int i = 0; i < nodes.size(); i++) {
+ names[i] = nodes.get(i).name;
+ }
+ return names;
+ }
+
+ /**
+ * Splits 100 percent into n two-decimal shares summing to exactly 100.00.
+ */
+ private static float[] splitPercentages(int n) {
+ float[] out = new float[n];
+ int base = 10000 / n;
Review Comment:
Nit: `generate(0)` (e.g. `-Dcs.bench.sizes=0`) fails here with `ArithmeticException: / by zero`. A `targetQueueCount > 0` check in `generate` would give a clearer error.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerConfigBenchmark.java:
##########
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.resourcemanager.MockRM;
+import org.apache.hadoop.yarn.server.resourcemanager.NodeAttributeTestUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
+import org.junit.jupiter.api.Test;
+import org.slf4j.event.Level;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Capacity Scheduler configuration benchmark: scheduler load, queue refresh
+ * and configuration validation on generated trees of
+ * {@link CSConfigBenchmarkGenerator}.
+ *
+ * <p>It only runs with {@code -DRunCapacitySchedulerConfigBenchmark=true}.
+ * Each size should run in its own JVM, and each fork is a separate Maven
+ * invocation, for example three forks of every size:</p>
+ * <pre>
+ * RM=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
+ * for fork in 1 2 3; do for size in 10 100 1000 5000; do
+ * ./mvnw -o test -pl $RM -Dtest=TestCapacitySchedulerConfigBenchmark \
+ * -DRunCapacitySchedulerConfigBenchmark=true -Dcs.bench.sizes=$size \
+ * -Dcs.bench.fork=$fork -Dcs.bench.out=/tmp/cs-bench.txt
+ * done; done
+ * </pre>
+ * <p>Properties:</p>
+ * <ul>
+ * <li>{@code cs.bench.sizes}: requested queue counts, default
+ * {@code 10,100,1000,5000};</li>
+ * <li>{@code cs.bench.warmups} (default 2) and {@code cs.bench.iterations}
+ * (default 5) per operation; the median of the iterations is reported;</li>
+ * <li>{@code cs.bench.ops}: subset of {@code scheduler-load},
+ * {@code refresh} and {@code validate}, default all;</li>
+ * <li>{@code cs.bench.validators}: implementations measured by the
+ * {@code validate} operation, default {@code legacy}
+ * ({@link CapacitySchedulerConfigValidator#validateCSConfiguration}, the
+ * core of {@code POST /scheduler-conf/validate}); any other entry is the
+ * class name of a {@link ProposalValidator} with a no-argument constructor,
+ * which is the hook for {@code validate/v2};</li>
+ * <li>{@code cs.bench.capacityMode}: a
+ * {@link CSConfigBenchmarkGenerator.CapacityMode}, default
+ * {@code PCT_WEIGHT_MIXED};</li>
+ * <li>{@code cs.bench.fork}: label printed with the results;</li>
+ * <li>{@code cs.bench.out}: file the {@code BENCH} result lines are
+ * appended to (they are always printed to standard output).</li>
+ * </ul>
+ * <p>Operations: {@code scheduler-load} initializes a fresh
+ * {@link CapacityScheduler} from the configuration, {@code refresh}
+ * reinitializes the running scheduler with the unchanged configuration (what
+ * {@code AdminService.refreshQueues} does after loading the file), and
+ * {@code validate} validates a proposal that moves one percent of capacity
+ * between two sibling leaves.</p>
+ */
+public class TestCapacitySchedulerConfigBenchmark {
+
+ /** A validation implementation measured by the {@code validate} op. */
+ public interface ProposalValidator {
+ /**
+ * Validates {@code proposed} (the live configuration with the proposal
+ * applied) against the running scheduler, throwing when it is invalid.
+ */
+ void validate(RMContext rmContext, Configuration live,
+ Configuration proposed) throws Exception;
+ }
+
+ /** The legacy {@code POST /scheduler-conf/validate} core. */
+ public static final class LegacyValidator implements ProposalValidator {
+ @Override
+ public void validate(RMContext rmContext, Configuration live,
+ Configuration proposed) throws IOException {
+ CapacitySchedulerConfigValidator.validateCSConfiguration(live, proposed,
+ rmContext);
+ }
+ }
+
+ private interface Op {
+ void run() throws Exception;
+ }
+
+ private final List<String> ops = Arrays.asList(System.getProperty(
+ "cs.bench.ops", "scheduler-load,refresh,validate").split(","));
+ private final int warmups = Integer.getInteger("cs.bench.warmups", 2);
+ private final int iterations = Integer.getInteger("cs.bench.iterations", 5);
+ private final String fork = System.getProperty("cs.bench.fork", "1");
+
+ @Test
+ public void testConfigurationOperations() throws Exception {
+ assumeTrue(Boolean.getBoolean("RunCapacitySchedulerConfigBenchmark"));
+ for (String size : System.getProperty("cs.bench.sizes", "10,100,1000,5000")
+ .split(",")) {
+ runForSize(Integer.parseInt(size.trim()));
+ }
+ }
+
+ private void runForSize(int requested) throws Exception {
+ CSConfigBenchmarkGenerator.GeneratedConfig generated =
+ CSConfigBenchmarkGenerator.generate(requested,
+ CSConfigBenchmarkGenerator.CapacityMode.valueOf(System.getProperty(
+ "cs.bench.capacityMode", "PCT_WEIGHT_MIXED")));
+ Map<String, String> proposal = toMap(
+ CSConfigBenchmarkGenerator.createMutatedCopy(generated.getConf(), generated));
+ YarnConfiguration conf =
+ NodeAttributeTestUtils.getRandomDirConf(generated.getConf());
+ conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
+ ResourceScheduler.class);
+
+ MockRM rm = new MockRM(conf);
+ try {
+ rm.start();
+ // MockRM switches the root logger to DEBUG, which dominates the timings.
+ GenericTestUtils.setRootLogLevel(Level.WARN);
+ rm.registerNode("h1:1234", 100 * 1024, 100);
+ CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler();
+ RMContext rmContext = rm.getRMContext();
+ int queues = cs.getCapacitySchedulerQueueManager().getQueues().size() - 1;
+ assertEquals(generated.getQueueCount(), queues);
+
+ measure(requested, queues, "scheduler-load", () -> {
+ CapacityScheduler fresh = new CapacityScheduler();
+ try {
+ fresh.setConf(cs.getConf());
+ fresh.setRMContext(rmContext);
+ fresh.init(cs.getConf());
+ assertEquals(queues + 1,
+ fresh.getCapacitySchedulerQueueManager().getQueues().size());
+ } finally {
+ fresh.stop();
+ }
+ });
+ measure(requested, queues, "refresh",
+ () -> cs.reinitialize(cs.getConf(), rmContext));
+
+ Configuration live = cs.getConf();
+ Configuration proposed = new Configuration(live);
+ proposal.forEach(proposed::set);
+ for (String name : System.getProperty("cs.bench.validators", "legacy")
+ .split(",")) {
+ ProposalValidator validator = name.equals("legacy") ? new LegacyValidator()
+ : (ProposalValidator) Class.forName(name.trim()).getDeclaredConstructor()
+ .newInstance();
+ measure(requested, queues, "validate", name.trim(),
+ () -> validator.validate(rmContext, live, proposed));
+ }
+ } finally {
+ rm.stop();
+ QueueMetrics.clearQueueMetrics();
+ GenericTestUtils.setRootLogLevel(Level.INFO);
+ }
+ }
+
+ private void measure(int requested, int queues, String op, Op body)
+ throws Exception {
+ measure(requested, queues, op, null, body);
+ }
+
+ private void measure(int requested, int queues, String op, String variant,
+ Op body) throws Exception {
+ if (!ops.contains(op)) {
+ return;
+ }
+ for (int i = 0; i < warmups; i++) {
+ body.run();
+ }
+ double[] samples = new double[iterations];
+ for (int i = 0; i < iterations; i++) {
+ long start = System.nanoTime();
+ body.run();
+ samples[i] = (System.nanoTime() - start) / 1e6;
+ }
+ double[] sorted = samples.clone();
+ Arrays.sort(sorted);
+ List<String> all = new ArrayList<>();
+ for (double sample : samples) {
+ all.add(String.format(Locale.ROOT, "%.1f", sample));
+ }
+ String line = String.format(Locale.ROOT,
+ "BENCH fork=%s requested=%d queues=%d op=%s%s warmups=%d n=%d "
+ + "median_ms=%.1f all_ms=%s",
+ fork, requested, queues, op, variant == null ? "" : "-" + variant,
+ warmups, iterations, sorted[(iterations - 1) / 2], String.join(",", all));
Review Comment:
`sorted[(iterations - 1) / 2]` is the lower middle sample when `cs.bench.iterations` is even, and throws `ArrayIndexOutOfBoundsException` for 0. Consider averaging the two middle samples for even counts and rejecting iterations < 1.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSConfigBenchmarkGenerator.java:
##########
@@ -0,0 +1,543 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+
+import static org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableSet.of;
+
+/**
+ * Generates synthetic but structurally realistic capacity-scheduler
+ * configuration content for a requested queue count.
+ * It feeds {@link TestCapacitySchedulerConfigBenchmark} (scheduler load,
+ * refresh and validation timings) and {@link TestCSConfigBenchmarkGenerator}.
+ *
+ * Tree shape (mixed depth 3 and 4):
+ * <pre>
+ * root
+ * p0 (pct children, node label, one legacy managed parent leaf)
+ * p1 (weight children, AQC v2 template on sub-parent s0)
+ * p2 (pct children, unlabeled)
+ * p3 (weight children)
+ * ...
+ * each pI: 2 direct leaves + up to 5 sub-parents with up to 9 leaves each
+ * </pre>
+ *
+ * Capacity modes:
+ * - PCT_WEIGHT_MIXED (default): root children are percentage; even-indexed
+ * subtrees use percentage children, odd-indexed use weight children; the last
+ * sub-parent of a weight subtree flips its children back to percentage
+ * (legal per the legacy-mode parent/child mode table in AbstractParentQueue).
+ * - ALL_ABSOLUTE: every queue uses absolute [memory=..,vcores=..] resources,
+ * with parent minimums equal to the sum of their children. This is a separate
+ * mode because in legacy queue mode absolute resources cannot be mixed with
+ * percentage/weight anywhere in one hierarchy (absolute children require an
+ * absolute parent, and siblings must be mode-uniform).
+ *
+ * Extras sprinkled for realism: accessible-node-labels + per-label capacities
+ * on every fourth subtree (labels "blue"/"ssd"), legacy managed-parent
+ * (auto-create-child-queue) leaf templates, auto-queue-creation-v2 templates,
+ * legacy queue-mapping placement rules, ACLs, user-limit-factor,
+ * maximum-capacity and ordering policies on a subset of queues.
+ */
+public final class CSConfigBenchmarkGenerator {
+
+ public enum CapacityMode {
+ PCT_WEIGHT_MIXED,
+ ALL_ABSOLUTE
+ }
+
+ /** Everything the harness needs to know about a generated config. */
+ public static final class GeneratedConfig {
+ private final CapacitySchedulerConfiguration conf;
+ private final int queueCount;
+ private final List<String> leafPaths;
+ private final String mutationLeafA;
+ private final String mutationLeafB;
+ private final List<String> labels;
+
+ private GeneratedConfig(CapacitySchedulerConfiguration conf, int queueCount,
+ List<String> leafPaths, String mutationLeafA, String mutationLeafB,
+ List<String> labels) {
+ this.conf = conf;
+ this.queueCount = queueCount;
+ this.leafPaths = leafPaths;
+ this.mutationLeafA = mutationLeafA;
+ this.mutationLeafB = mutationLeafB;
+ this.labels = labels;
+ }
+
+ public CapacitySchedulerConfiguration getConf() {
+ return conf;
+ }
+
+ /** Number of generated queues, root excluded. */
+ public int getQueueCount() {
+ return queueCount;
+ }
+
+ public List<String> getLeafPaths() {
+ return Collections.unmodifiableList(leafPaths);
+ }
+
+ /** Two sibling leaves whose capacities can be swapped for a valid change. */
+ public String getMutationLeafA() {
+ return mutationLeafA;
+ }
+
+ public String getMutationLeafB() {
+ return mutationLeafB;
+ }
+
+ public List<String> getLabels() {
+ return Collections.unmodifiableList(labels);
+ }
+ }
+
+ private static final int DIRECT_LEAVES = 2;
+ private static final int SUB_PARENTS = 5;
+ private static final int LEAVES_PER_SUB_PARENT = 9;
+ private static final int MAX_MANAGED_PARENTS = 4;
+ private static final int MAX_AQC_V2_PARENTS = 4;
+ private static final String LABEL_BLUE = "blue";
+ private static final String LABEL_SSD = "ssd";
+ private static final String CS_PREFIX = CapacitySchedulerConfiguration.PREFIX;
+
+ private CSConfigBenchmarkGenerator() {
+ }
+
+ private static final class Node {
+ private final String name;
+ private final String path;
+ private final List<Node> children = new ArrayList<>();
+ private boolean managedParent;
+
+ Node(String parentPath, String name) {
+ this.name = name;
+ this.path = parentPath + "." + name;
+ }
+
+ boolean isLeaf() {
+ return children.isEmpty() && !managedParent;
+ }
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount) {
+ return generate(targetQueueCount, CapacityMode.PCT_WEIGHT_MIXED);
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount, CapacityMode mode) {
+ List<Node> tops = buildForest(targetQueueCount);
+ CapacitySchedulerConfiguration conf =
+ new CapacitySchedulerConfiguration(new Configuration(false), false);
+
+ markManagedParents(tops);
+ QueuePath rootPath = new QueuePath(CapacitySchedulerConfiguration.ROOT);
+ conf.setQueues(rootPath, names(tops));
+
+ if (mode == CapacityMode.ALL_ABSOLUTE) {
+ emitAbsolute(conf, rootPath, tops);
+ } else {
+ emitMixed(conf, tops);
+ }
+
+ List<String> leafPaths = new ArrayList<>();
+ int count = 0;
+ for (Node top : tops) {
+ count += collect(top, leafPaths);
+ }
+ String[] mutationPair = pickMutationSiblings(tops);
+ setPlacementRules(conf, leafPaths);
+
+ List<String> labels = new ArrayList<>();
+ if (mode == CapacityMode.PCT_WEIGHT_MIXED) {
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null && !labels.contains(label)) {
+ labels.add(label);
+ }
+ }
+ }
+ return new GeneratedConfig(conf, count, leafPaths, mutationPair[0],
+ mutationPair[1], labels);
+ }
+
+ /**
+ * Returns a copy of {@code base} with a valid capacity change applied: one
+ * percent is moved from mutation leaf B to mutation leaf A (and likewise for
+ * their per-label capacities), so per-parent sums stay at 100.
+ */
+ public static Configuration createMutatedCopy(Configuration base,
+ GeneratedConfig gen) {
+ Configuration copy = new Configuration(base);
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), null);
+ for (String label : gen.getLabels()) {
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), label);
+ }
+ return copy;
+ }
+
+ private static void moveOnePercent(Configuration conf, String leafA,
+ String leafB, String label) {
+ if (leafA == null || leafB == null) {
+ return;
+ }
+ String suffix = (label == null) ? ".capacity"
+ : ".accessible-node-labels." + label + ".capacity";
+ String keyA = CS_PREFIX + leafA + suffix;
+ String keyB = CS_PREFIX + leafB + suffix;
+ String a = conf.get(keyA);
+ String b = conf.get(keyB);
+ if (a == null || b == null) {
+ return;
+ }
+ try {
+ conf.set(keyA, String.valueOf(Float.parseFloat(a) + 1.0f));
+ conf.set(keyB, String.valueOf(Float.parseFloat(b) - 1.0f));
+ } catch (NumberFormatException e) {
+ // Absolute/weight profiles mutate a real queue limit, never a no-op.
+ conf.set(CS_PREFIX + leafA + ".maximum-applications", "2345");
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Tree construction
+ // ---------------------------------------------------------------------
+
+ private static List<Node> buildForest(int targetQueueCount) {
+ List<Node> tops = new ArrayList<>();
+ int fullSubtree = 1 + DIRECT_LEAVES + SUB_PARENTS * (1 + LEAVES_PER_SUB_PARENT);
+ int remaining = targetQueueCount;
+ int i = 0;
+ while (remaining > 0) {
+ int budget = Math.min(fullSubtree, remaining);
+ Node top = new Node(CapacitySchedulerConfiguration.ROOT, "p" + i);
+ budget--;
+ int directLeaves = Math.min(DIRECT_LEAVES, budget);
+ for (int l = 0; l < directLeaves; l++) {
+ top.children.add(new Node(top.path, "l" + l));
+ }
+ budget -= directLeaves;
+ int s = 0;
+ while (budget > 0) {
+ Node sub = new Node(top.path, "s" + s);
+ top.children.add(sub);
+ budget--;
+ int subLeaves = Math.min(LEAVES_PER_SUB_PARENT, budget);
+ for (int l = 0; l < subLeaves; l++) {
+ sub.children.add(new Node(sub.path, "l" + l));
+ }
+ budget -= subLeaves;
+ s++;
+ }
+ remaining -= subtreeSize(top);
+ tops.add(top);
+ i++;
+ }
+ return tops;
+ }
+
+ private static int subtreeSize(Node node) {
+ int size = 1;
+ for (Node child : node.children) {
+ size += subtreeSize(child);
+ }
+ return size;
+ }
+
+ /**
+ * Converts the last direct leaf of every other percentage subtree into a
+ * legacy managed parent (auto-create-child-queue.enabled), capped globally.
+ */
+ private static void markManagedParents(List<Node> tops) {
+ int marked = 0;
+ for (int i = 0; i < tops.size() && marked < MAX_MANAGED_PARENTS; i += 4) {
+ Node top = tops.get(i);
+ for (int c = top.children.size() - 1; c >= 0; c--) {
+ Node child = top.children.get(c);
+ if (child.isLeaf()) {
+ child.managedParent = true;
+ marked++;
+ break;
+ }
+ }
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Mixed percentage/weight emission
+ // ---------------------------------------------------------------------
+
+ private static void emitMixed(CapacitySchedulerConfiguration conf, List<Node> tops) {
+ float[] topCaps = splitPercentages(tops.size());
+ Map<String, List<Integer>> labelMembers = new LinkedHashMap<>();
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null) {
+ labelMembers.computeIfAbsent(label, k -> new ArrayList<>()).add(i);
+ }
+ }
+ // Per-label percentage shares among the labeled root children; the sums per
+ // label are 100 among label-accessible siblings, as legacy mode requires.
+ Map<Integer, Float> topLabelCap = new LinkedHashMap<>();
+ for (List<Integer> members : labelMembers.values()) {
+ float[] shares = splitPercentages(members.size());
+ for (int m = 0; m < members.size(); m++) {
+ topLabelCap.put(members.get(m), shares[m]);
+ }
+ }
+
+ int aqcV2Marked = 0;
+ for (int i = 0; i < tops.size(); i++) {
+ Node top = tops.get(i);
+ QueuePath topPath = new QueuePath(top.path);
+ conf.setCapacity(topPath, topCaps[i]);
+ conf.setMaximumCapacity(topPath, 100f);
+ String label = labelFor(i);
+ if (label != null) {
+ conf.setAccessibleNodeLabels(topPath, of(label));
+ conf.setCapacityByLabel(topPath, label, topLabelCap.get(i));
+ }
+ boolean weightMode = (i % 2 == 1);
+ boolean markAqcV2 = weightMode && (i % 6 == 1) && aqcV2Marked < MAX_AQC_V2_PARENTS;
+ if (markAqcV2) {
+ aqcV2Marked++;
+ }
+ emitChildren(conf, top, weightMode, label, true, markAqcV2);
+ }
+ }
+
+ /**
+ * Emits capacities (and per-label capacities) for the children of
+ * {@code parent}, then recurses one level into sub-parents.
+ */
+ private static void emitChildren(CapacitySchedulerConfiguration conf, Node parent,
+ boolean weightMode, String label, boolean topLevel, boolean markAqcV2) {
+ if (parent.children.isEmpty()) {
+ return;
+ }
+ conf.setQueues(new QueuePath(parent.path), names(parent.children));
+ float[] caps = splitPercentages(parent.children.size());
+ for (int c = 0; c < parent.children.size(); c++) {
+ Node child = parent.children.get(c);
+ QueuePath childPath = new QueuePath(child.path);
+ // The last sub-parent of a top-level weight subtree flips its children
+ // back to percentage mode; mode must stay uniform among siblings only.
+ boolean childrenWeightMode = weightMode
+ && !(topLevel && c == parent.children.size() - 1 && !child.children.isEmpty());
+ if (weightMode) {
+ conf.setNonLabeledQueueWeight(childPath, (c % 4) + 1f);
+ } else {
+ conf.setCapacity(childPath, caps[c]);
+ conf.setMaximumCapacity(childPath, 100f);
+ }
+ if (label != null) {
+ conf.setAccessibleNodeLabels(childPath, of(label));
+ conf.setCapacityByLabel(childPath, label, caps[c]);
+ }
+ decorate(conf, child, c);
+ if (child.managedParent) {
+ emitManagedParent(conf, childPath, label);
+ } else if (!child.children.isEmpty()) {
+ if (markAqcV2 && c == DIRECT_LEAVES) {
Review Comment:
When a weight subtree has a single sub-parent (e.g. `generate(60)`, `root.p1`), `s0` is also the last sub-parent, so its children are flipped back to percentage, but it still gets the `2w` AQC v2 template. Creating a dynamic queue under it then fails:
```
SchedulerDynamicEditException: Trying to create new queue=root.p1.s0.dyn but not all the queues under parent=root.p1.s0 are using weight-based capacity
```
With `generate(200)`, `s0` has weight children and the same `createQueue` succeeds. Suggest `if (markAqcV2 && c == DIRECT_LEAVES && childrenWeightMode)`, and maybe an assertion in `TestCSConfigBenchmarkGenerator` that each AQC v2 parent can create a queue.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java:
##########
@@ -637,6 +642,73 @@ public void testComplexValidateAbsoluteResourceConfig() throws Exception {
rm.stop();
}
+ @Test
+ public void testQueueMixingAbsoluteAndPercentageAcrossLabelsRejected() {
+ CapacitySchedulerConfiguration csConf = new CapacitySchedulerConfiguration();
+ csConf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
+ ResourceScheduler.class);
+ csConf.setQueues(ROOT, new String[]{QUEUEA, QUEUEB});
+ csConf.setCapacity(QUEUEA_FULL, 50f);
+ csConf.setCapacity(QUEUEB_FULL, 50f);
+ csConf.setAccessibleNodeLabels(QUEUEA_FULL, ImmutableSet.of(X_LABEL));
+ csConf.setMinimumResourceRequirement(X_LABEL, QUEUEA_FULL, Resource.newInstance(GB, 1));
+
+ Exception e = assertThrows(Exception.class, () -> {
+ try (MockRM rm = new MockRM(csConf)) {
+ rm.start();
+ }
+ });
+ assertEquals("Queue 'root.queueA' should use either percentage based capacity "
+ + "configuration or absolute resource.", ExceptionUtils.getRootCause(e).getMessage());
+ }
+
+ @Test
+ public void testDefaultResourceCalculatorAcceptsChildVcoresAboveParent()
+ throws Exception {
+ CapacitySchedulerConfiguration csConf = new CapacitySchedulerConfiguration();
+ csConf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
+ ResourceScheduler.class);
+ csConf.setQueues(ROOT, new String[]{QUEUEA, QUEUEB});
+ csConf.setQueues(QUEUEA_FULL, new String[]{QUEUEA1, QUEUEA2});
+ csConf.setCapacity(QUEUEA_FULL, "[memory=40960,vcores=4]");
+ csConf.setCapacity(QUEUEA1_FULL, "[memory=20480,vcores=8]");
+ csConf.setCapacity(QUEUEA2_FULL, "[memory=20Gi,vcores=8]");
+ csConf.setCapacity(QUEUEB_FULL, "[memory=40960,vcores=40]");
+
+ try (MockRM rm = new MockRM(csConf)) {
+ rm.start();
+ CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler();
+ // Only memory is compared against the parent, so 16 vcores fit under 4.
+ assertEquals(Resource.newInstance(20 * GB, 8), cs.getQueue(QUEUEA1_FULL.getFullPath())
+ .getQueueResourceQuotas().getConfiguredMinResource());
+ assertEquals(Resource.newInstance(20 * GB, 8), cs.getQueue(QUEUEA2_FULL.getFullPath())
+ .getQueueResourceQuotas().getConfiguredMinResource());
+ }
+ }
+
+ @Test
+ public void testDominantResourceCalculatorScalesOversubscribedQueues()
+ throws Exception {
+ CapacitySchedulerConfiguration csConf = new CapacitySchedulerConfiguration();
+ csConf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
+ ResourceScheduler.class);
+ csConf.setResourceComparator(DominantResourceCalculator.class);
+ csConf.setQueues(ROOT, new String[]{QUEUEA, QUEUEB});
+ csConf.setCapacity(QUEUEA_FULL, "[memory=10240,vcores=60]");
+ csConf.setCapacity(QUEUEB_FULL, "[memory=10240,vcores=60]");
+
+ try (MockRM rm = new MockRM(csConf)) {
+ rm.start();
+ rm.registerNode("127.0.0.1:1234", 100 * GB, 100);
+ CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler();
+ // 120 configured vcores exceed the cluster, so both queues are scaled down.
Review Comment:
Nit: memory actually goes up here (10 GB configured, 50 GB effective) while vcores go down, so "scaled down" is a bit misleading. Maybe: "each queue gets half of the cluster in every resource".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#discussion_r4237383195
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerConfigBenchmark.java:
##########
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.resourcemanager.MockRM;
+import org.apache.hadoop.yarn.server.resourcemanager.NodeAttributeTestUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
+import org.junit.jupiter.api.Test;
+import org.slf4j.event.Level;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Capacity Scheduler configuration benchmark: scheduler load, queue refresh
+ * and configuration validation on generated trees of
+ * {@link CSConfigBenchmarkGenerator}.
+ *
+ * <p>It only runs with {@code -DRunCapacitySchedulerConfigBenchmark=true}.
+ * Each size should run in its own JVM, and each fork is a separate Maven
+ * invocation, for example three forks of every size:</p>
+ * <pre>
+ * RM=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
+ * for fork in 1 2 3; do for size in 10 100 1000 5000; do
+ * ./mvnw -o test -pl $RM -Dtest=TestCapacitySchedulerConfigBenchmark \
+ * -DRunCapacitySchedulerConfigBenchmark=true -Dcs.bench.sizes=$size \
+ * -Dcs.bench.fork=$fork -Dcs.bench.out=/tmp/cs-bench.txt
+ * done; done
+ * </pre>
+ * <p>Properties:</p>
+ * <ul>
+ * <li>{@code cs.bench.sizes}: requested queue counts, default
+ * {@code 10,100,1000,5000};</li>
+ * <li>{@code cs.bench.warmups} (default 2) and {@code cs.bench.iterations}
+ * (default 5) per operation; the median of the iterations is reported;</li>
+ * <li>{@code cs.bench.ops}: subset of {@code scheduler-load},
+ * {@code refresh} and {@code validate}, default all;</li>
+ * <li>{@code cs.bench.validators}: implementations measured by the
+ * {@code validate} operation, default {@code legacy}
+ * ({@link CapacitySchedulerConfigValidator#validateCSConfiguration}, the
+ * core of {@code POST /scheduler-conf/validate}); any other entry is the
+ * class name of a {@link ProposalValidator} with a no-argument constructor,
+ * which is the hook for {@code validate/v2};</li>
+ * <li>{@code cs.bench.capacityMode}: a
+ * {@link CSConfigBenchmarkGenerator.CapacityMode}, default
+ * {@code PCT_WEIGHT_MIXED};</li>
+ * <li>{@code cs.bench.fork}: label printed with the results;</li>
+ * <li>{@code cs.bench.out}: file the {@code BENCH} result lines are
+ * appended to (they are always printed to standard output).</li>
+ * </ul>
+ * <p>Operations: {@code scheduler-load} initializes a fresh
+ * {@link CapacityScheduler} from the configuration, {@code refresh}
+ * reinitializes the running scheduler with the unchanged configuration (what
+ * {@code AdminService.refreshQueues} does after loading the file), and
+ * {@code validate} validates a proposal that moves one percent of capacity
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#issuecomment-6041769326
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 30 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 31m 25s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 31m 57s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 56s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 551 unchanged - 2 fixed = 552 total (was 553) |
| +1 :green_heart: | mvnsite | 1m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 111m 19s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 240m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/2/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux dc5a9fda63e3 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 768188c4128e47a8db6ab5fed3846c96a23cf83b |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/2/testReport/ |
| Max. process+thread count | 969 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#discussion_r4237382604
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSConfigBenchmarkGenerator.java:
##########
@@ -0,0 +1,543 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+
+import static org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableSet.of;
+
+/**
+ * Generates synthetic but structurally realistic capacity-scheduler
+ * configuration content for a requested queue count.
+ * It feeds {@link TestCapacitySchedulerConfigBenchmark} (scheduler load,
+ * refresh and validation timings) and {@link TestCSConfigBenchmarkGenerator}.
+ *
+ * Tree shape (mixed depth 3 and 4):
+ * <pre>
+ * root
+ * p0 (pct children, node label, one legacy managed parent leaf)
+ * p1 (weight children, AQC v2 template on sub-parent s0)
+ * p2 (pct children, unlabeled)
+ * p3 (weight children)
+ * ...
+ * each pI: 2 direct leaves + up to 5 sub-parents with up to 9 leaves each
+ * </pre>
+ *
+ * Capacity modes:
+ * - PCT_WEIGHT_MIXED (default): root children are percentage; even-indexed
+ * subtrees use percentage children, odd-indexed use weight children; the last
+ * sub-parent of a weight subtree flips its children back to percentage
+ * (legal per the legacy-mode parent/child mode table in AbstractParentQueue).
+ * - ALL_ABSOLUTE: every queue uses absolute [memory=..,vcores=..] resources,
+ * with parent minimums equal to the sum of their children. This is a separate
+ * mode because in legacy queue mode absolute resources cannot be mixed with
+ * percentage/weight anywhere in one hierarchy (absolute children require an
+ * absolute parent, and siblings must be mode-uniform).
+ *
+ * Extras sprinkled for realism: accessible-node-labels + per-label capacities
+ * on every fourth subtree (labels "blue"/"ssd"), legacy managed-parent
+ * (auto-create-child-queue) leaf templates, auto-queue-creation-v2 templates,
+ * legacy queue-mapping placement rules, ACLs, user-limit-factor,
+ * maximum-capacity and ordering policies on a subset of queues.
+ */
+public final class CSConfigBenchmarkGenerator {
+
+ public enum CapacityMode {
+ PCT_WEIGHT_MIXED,
+ ALL_ABSOLUTE
+ }
+
+ /** Everything the harness needs to know about a generated config. */
+ public static final class GeneratedConfig {
+ private final CapacitySchedulerConfiguration conf;
+ private final int queueCount;
+ private final List<String> leafPaths;
+ private final String mutationLeafA;
+ private final String mutationLeafB;
+ private final List<String> labels;
+
+ private GeneratedConfig(CapacitySchedulerConfiguration conf, int queueCount,
+ List<String> leafPaths, String mutationLeafA, String mutationLeafB,
+ List<String> labels) {
+ this.conf = conf;
+ this.queueCount = queueCount;
+ this.leafPaths = leafPaths;
+ this.mutationLeafA = mutationLeafA;
+ this.mutationLeafB = mutationLeafB;
+ this.labels = labels;
+ }
+
+ public CapacitySchedulerConfiguration getConf() {
+ return conf;
+ }
+
+ /** Number of generated queues, root excluded. */
+ public int getQueueCount() {
+ return queueCount;
+ }
+
+ public List<String> getLeafPaths() {
+ return Collections.unmodifiableList(leafPaths);
+ }
+
+ /** Two sibling leaves whose capacities can be swapped for a valid change. */
+ public String getMutationLeafA() {
+ return mutationLeafA;
+ }
+
+ public String getMutationLeafB() {
+ return mutationLeafB;
+ }
+
+ public List<String> getLabels() {
+ return Collections.unmodifiableList(labels);
+ }
+ }
+
+ private static final int DIRECT_LEAVES = 2;
+ private static final int SUB_PARENTS = 5;
+ private static final int LEAVES_PER_SUB_PARENT = 9;
+ private static final int MAX_MANAGED_PARENTS = 4;
+ private static final int MAX_AQC_V2_PARENTS = 4;
+ private static final String LABEL_BLUE = "blue";
+ private static final String LABEL_SSD = "ssd";
+ private static final String CS_PREFIX = CapacitySchedulerConfiguration.PREFIX;
+
+ private CSConfigBenchmarkGenerator() {
+ }
+
+ private static final class Node {
+ private final String name;
+ private final String path;
+ private final List<Node> children = new ArrayList<>();
+ private boolean managedParent;
+
+ Node(String parentPath, String name) {
+ this.name = name;
+ this.path = parentPath + "." + name;
+ }
+
+ boolean isLeaf() {
+ return children.isEmpty() && !managedParent;
+ }
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount) {
+ return generate(targetQueueCount, CapacityMode.PCT_WEIGHT_MIXED);
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount, CapacityMode mode) {
+ List<Node> tops = buildForest(targetQueueCount);
+ CapacitySchedulerConfiguration conf =
+ new CapacitySchedulerConfiguration(new Configuration(false), false);
+
+ markManagedParents(tops);
+ QueuePath rootPath = new QueuePath(CapacitySchedulerConfiguration.ROOT);
+ conf.setQueues(rootPath, names(tops));
+
+ if (mode == CapacityMode.ALL_ABSOLUTE) {
+ emitAbsolute(conf, rootPath, tops);
+ } else {
+ emitMixed(conf, tops);
+ }
+
+ List<String> leafPaths = new ArrayList<>();
+ int count = 0;
+ for (Node top : tops) {
+ count += collect(top, leafPaths);
+ }
+ String[] mutationPair = pickMutationSiblings(tops);
+ setPlacementRules(conf, leafPaths);
+
+ List<String> labels = new ArrayList<>();
+ if (mode == CapacityMode.PCT_WEIGHT_MIXED) {
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null && !labels.contains(label)) {
+ labels.add(label);
+ }
+ }
+ }
+ return new GeneratedConfig(conf, count, leafPaths, mutationPair[0],
+ mutationPair[1], labels);
+ }
+
+ /**
+ * Returns a copy of {@code base} with a valid capacity change applied: one
+ * percent is moved from mutation leaf B to mutation leaf A (and likewise for
+ * their per-label capacities), so per-parent sums stay at 100.
+ */
+ public static Configuration createMutatedCopy(Configuration base,
+ GeneratedConfig gen) {
+ Configuration copy = new Configuration(base);
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), null);
+ for (String label : gen.getLabels()) {
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), label);
+ }
+ return copy;
+ }
+
+ private static void moveOnePercent(Configuration conf, String leafA,
+ String leafB, String label) {
+ if (leafA == null || leafB == null) {
+ return;
+ }
+ String suffix = (label == null) ? ".capacity"
+ : ".accessible-node-labels." + label + ".capacity";
+ String keyA = CS_PREFIX + leafA + suffix;
+ String keyB = CS_PREFIX + leafB + suffix;
+ String a = conf.get(keyA);
+ String b = conf.get(keyB);
+ if (a == null || b == null) {
+ return;
+ }
+ try {
+ conf.set(keyA, String.valueOf(Float.parseFloat(a) + 1.0f));
+ conf.set(keyB, String.valueOf(Float.parseFloat(b) - 1.0f));
+ } catch (NumberFormatException e) {
+ // Absolute/weight profiles mutate a real queue limit, never a no-op.
+ conf.set(CS_PREFIX + leafA + ".maximum-applications", "2345");
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Tree construction
+ // ---------------------------------------------------------------------
+
+ private static List<Node> buildForest(int targetQueueCount) {
+ List<Node> tops = new ArrayList<>();
+ int fullSubtree = 1 + DIRECT_LEAVES + SUB_PARENTS * (1 + LEAVES_PER_SUB_PARENT);
+ int remaining = targetQueueCount;
+ int i = 0;
+ while (remaining > 0) {
+ int budget = Math.min(fullSubtree, remaining);
+ Node top = new Node(CapacitySchedulerConfiguration.ROOT, "p" + i);
+ budget--;
+ int directLeaves = Math.min(DIRECT_LEAVES, budget);
+ for (int l = 0; l < directLeaves; l++) {
+ top.children.add(new Node(top.path, "l" + l));
+ }
+ budget -= directLeaves;
+ int s = 0;
+ while (budget > 0) {
+ Node sub = new Node(top.path, "s" + s);
+ top.children.add(sub);
+ budget--;
+ int subLeaves = Math.min(LEAVES_PER_SUB_PARENT, budget);
+ for (int l = 0; l < subLeaves; l++) {
+ sub.children.add(new Node(sub.path, "l" + l));
+ }
+ budget -= subLeaves;
+ s++;
+ }
+ remaining -= subtreeSize(top);
+ tops.add(top);
+ i++;
+ }
+ return tops;
+ }
+
+ private static int subtreeSize(Node node) {
+ int size = 1;
+ for (Node child : node.children) {
+ size += subtreeSize(child);
+ }
+ return size;
+ }
+
+ /**
+ * Converts the last direct leaf of every other percentage subtree into a
+ * legacy managed parent (auto-create-child-queue.enabled), capped globally.
+ */
+ private static void markManagedParents(List<Node> tops) {
+ int marked = 0;
+ for (int i = 0; i < tops.size() && marked < MAX_MANAGED_PARENTS; i += 4) {
+ Node top = tops.get(i);
+ for (int c = top.children.size() - 1; c >= 0; c--) {
+ Node child = top.children.get(c);
+ if (child.isLeaf()) {
+ child.managedParent = true;
+ marked++;
+ break;
+ }
+ }
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Mixed percentage/weight emission
+ // ---------------------------------------------------------------------
+
+ private static void emitMixed(CapacitySchedulerConfiguration conf, List<Node> tops) {
+ float[] topCaps = splitPercentages(tops.size());
+ Map<String, List<Integer>> labelMembers = new LinkedHashMap<>();
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null) {
+ labelMembers.computeIfAbsent(label, k -> new ArrayList<>()).add(i);
+ }
+ }
+ // Per-label percentage shares among the labeled root children; the sums per
+ // label are 100 among label-accessible siblings, as legacy mode requires.
+ Map<Integer, Float> topLabelCap = new LinkedHashMap<>();
+ for (List<Integer> members : labelMembers.values()) {
+ float[] shares = splitPercentages(members.size());
+ for (int m = 0; m < members.size(); m++) {
+ topLabelCap.put(members.get(m), shares[m]);
+ }
+ }
+
+ int aqcV2Marked = 0;
+ for (int i = 0; i < tops.size(); i++) {
+ Node top = tops.get(i);
+ QueuePath topPath = new QueuePath(top.path);
+ conf.setCapacity(topPath, topCaps[i]);
+ conf.setMaximumCapacity(topPath, 100f);
+ String label = labelFor(i);
+ if (label != null) {
+ conf.setAccessibleNodeLabels(topPath, of(label));
+ conf.setCapacityByLabel(topPath, label, topLabelCap.get(i));
+ }
+ boolean weightMode = (i % 2 == 1);
+ boolean markAqcV2 = weightMode && (i % 6 == 1) && aqcV2Marked < MAX_AQC_V2_PARENTS;
+ if (markAqcV2) {
+ aqcV2Marked++;
+ }
+ emitChildren(conf, top, weightMode, label, true, markAqcV2);
+ }
+ }
+
+ /**
+ * Emits capacities (and per-label capacities) for the children of
+ * {@code parent}, then recurses one level into sub-parents.
+ */
+ private static void emitChildren(CapacitySchedulerConfiguration conf, Node parent,
+ boolean weightMode, String label, boolean topLevel, boolean markAqcV2) {
+ if (parent.children.isEmpty()) {
+ return;
+ }
+ conf.setQueues(new QueuePath(parent.path), names(parent.children));
+ float[] caps = splitPercentages(parent.children.size());
+ for (int c = 0; c < parent.children.size(); c++) {
+ Node child = parent.children.get(c);
+ QueuePath childPath = new QueuePath(child.path);
+ // The last sub-parent of a top-level weight subtree flips its children
+ // back to percentage mode; mode must stay uniform among siblings only.
+ boolean childrenWeightMode = weightMode
+ && !(topLevel && c == parent.children.size() - 1 && !child.children.isEmpty());
+ if (weightMode) {
+ conf.setNonLabeledQueueWeight(childPath, (c % 4) + 1f);
+ } else {
+ conf.setCapacity(childPath, caps[c]);
+ conf.setMaximumCapacity(childPath, 100f);
+ }
+ if (label != null) {
+ conf.setAccessibleNodeLabels(childPath, of(label));
+ conf.setCapacityByLabel(childPath, label, caps[c]);
+ }
+ decorate(conf, child, c);
+ if (child.managedParent) {
+ emitManagedParent(conf, childPath, label);
+ } else if (!child.children.isEmpty()) {
+ if (markAqcV2 && c == DIRECT_LEAVES) {
Review Comment:
Good catch, thanks. Updated.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerConfigBenchmark.java:
##########
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.resourcemanager.MockRM;
+import org.apache.hadoop.yarn.server.resourcemanager.NodeAttributeTestUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
+import org.junit.jupiter.api.Test;
+import org.slf4j.event.Level;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Capacity Scheduler configuration benchmark: scheduler load, queue refresh
+ * and configuration validation on generated trees of
+ * {@link CSConfigBenchmarkGenerator}.
+ *
+ * <p>It only runs with {@code -DRunCapacitySchedulerConfigBenchmark=true}.
+ * Each size should run in its own JVM, and each fork is a separate Maven
+ * invocation, for example three forks of every size:</p>
+ * <pre>
+ * RM=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
+ * for fork in 1 2 3; do for size in 10 100 1000 5000; do
+ * ./mvnw -o test -pl $RM -Dtest=TestCapacitySchedulerConfigBenchmark \
+ * -DRunCapacitySchedulerConfigBenchmark=true -Dcs.bench.sizes=$size \
+ * -Dcs.bench.fork=$fork -Dcs.bench.out=/tmp/cs-bench.txt
+ * done; done
+ * </pre>
+ * <p>Properties:</p>
+ * <ul>
+ * <li>{@code cs.bench.sizes}: requested queue counts, default
+ * {@code 10,100,1000,5000};</li>
+ * <li>{@code cs.bench.warmups} (default 2) and {@code cs.bench.iterations}
+ * (default 5) per operation; the median of the iterations is reported;</li>
+ * <li>{@code cs.bench.ops}: subset of {@code scheduler-load},
+ * {@code refresh} and {@code validate}, default all;</li>
+ * <li>{@code cs.bench.validators}: implementations measured by the
+ * {@code validate} operation, default {@code legacy}
+ * ({@link CapacitySchedulerConfigValidator#validateCSConfiguration}, the
+ * core of {@code POST /scheduler-conf/validate}); any other entry is the
+ * class name of a {@link ProposalValidator} with a no-argument constructor,
+ * which is the hook for {@code validate/v2};</li>
+ * <li>{@code cs.bench.capacityMode}: a
+ * {@link CSConfigBenchmarkGenerator.CapacityMode}, default
+ * {@code PCT_WEIGHT_MIXED};</li>
+ * <li>{@code cs.bench.fork}: label printed with the results;</li>
+ * <li>{@code cs.bench.out}: file the {@code BENCH} result lines are
+ * appended to (they are always printed to standard output).</li>
+ * </ul>
+ * <p>Operations: {@code scheduler-load} initializes a fresh
+ * {@link CapacityScheduler} from the configuration, {@code refresh}
+ * reinitializes the running scheduler with the unchanged configuration (what
+ * {@code AdminService.refreshQueues} does after loading the file), and
+ * {@code validate} validates a proposal that moves one percent of capacity
+ * between two sibling leaves.</p>
+ */
+public class TestCapacitySchedulerConfigBenchmark {
+
+ /** A validation implementation measured by the {@code validate} op. */
+ public interface ProposalValidator {
+ /**
+ * Validates {@code proposed} (the live configuration with the proposal
+ * applied) against the running scheduler, throwing when it is invalid.
+ */
+ void validate(RMContext rmContext, Configuration live,
+ Configuration proposed) throws Exception;
+ }
+
+ /** The legacy {@code POST /scheduler-conf/validate} core. */
+ public static final class LegacyValidator implements ProposalValidator {
+ @Override
+ public void validate(RMContext rmContext, Configuration live,
+ Configuration proposed) throws IOException {
+ CapacitySchedulerConfigValidator.validateCSConfiguration(live, proposed,
+ rmContext);
+ }
+ }
+
+ private interface Op {
+ void run() throws Exception;
+ }
+
+ private final List<String> ops = Arrays.asList(System.getProperty(
+ "cs.bench.ops", "scheduler-load,refresh,validate").split(","));
+ private final int warmups = Integer.getInteger("cs.bench.warmups", 2);
+ private final int iterations = Integer.getInteger("cs.bench.iterations", 5);
+ private final String fork = System.getProperty("cs.bench.fork", "1");
+
+ @Test
+ public void testConfigurationOperations() throws Exception {
+ assumeTrue(Boolean.getBoolean("RunCapacitySchedulerConfigBenchmark"));
+ for (String size : System.getProperty("cs.bench.sizes", "10,100,1000,5000")
+ .split(",")) {
+ runForSize(Integer.parseInt(size.trim()));
+ }
+ }
+
+ private void runForSize(int requested) throws Exception {
+ CSConfigBenchmarkGenerator.GeneratedConfig generated =
+ CSConfigBenchmarkGenerator.generate(requested,
+ CSConfigBenchmarkGenerator.CapacityMode.valueOf(System.getProperty(
+ "cs.bench.capacityMode", "PCT_WEIGHT_MIXED")));
+ Map<String, String> proposal = toMap(
+ CSConfigBenchmarkGenerator.createMutatedCopy(generated.getConf(), generated));
+ YarnConfiguration conf =
+ NodeAttributeTestUtils.getRandomDirConf(generated.getConf());
+ conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
+ ResourceScheduler.class);
+
+ MockRM rm = new MockRM(conf);
+ try {
+ rm.start();
+ // MockRM switches the root logger to DEBUG, which dominates the timings.
+ GenericTestUtils.setRootLogLevel(Level.WARN);
+ rm.registerNode("h1:1234", 100 * 1024, 100);
+ CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler();
+ RMContext rmContext = rm.getRMContext();
+ int queues = cs.getCapacitySchedulerQueueManager().getQueues().size() - 1;
+ assertEquals(generated.getQueueCount(), queues);
+
+ measure(requested, queues, "scheduler-load", () -> {
+ CapacityScheduler fresh = new CapacityScheduler();
+ try {
+ fresh.setConf(cs.getConf());
+ fresh.setRMContext(rmContext);
+ fresh.init(cs.getConf());
+ assertEquals(queues + 1,
+ fresh.getCapacitySchedulerQueueManager().getQueues().size());
+ } finally {
+ fresh.stop();
+ }
+ });
+ measure(requested, queues, "refresh",
+ () -> cs.reinitialize(cs.getConf(), rmContext));
+
+ Configuration live = cs.getConf();
+ Configuration proposed = new Configuration(live);
+ proposal.forEach(proposed::set);
+ for (String name : System.getProperty("cs.bench.validators", "legacy")
+ .split(",")) {
+ ProposalValidator validator = name.equals("legacy") ? new LegacyValidator()
+ : (ProposalValidator) Class.forName(name.trim()).getDeclaredConstructor()
+ .newInstance();
+ measure(requested, queues, "validate", name.trim(),
+ () -> validator.validate(rmContext, live, proposed));
+ }
+ } finally {
+ rm.stop();
+ QueueMetrics.clearQueueMetrics();
+ GenericTestUtils.setRootLogLevel(Level.INFO);
+ }
+ }
+
+ private void measure(int requested, int queues, String op, Op body)
+ throws Exception {
+ measure(requested, queues, op, null, body);
+ }
+
+ private void measure(int requested, int queues, String op, String variant,
+ Op body) throws Exception {
+ if (!ops.contains(op)) {
+ return;
+ }
+ for (int i = 0; i < warmups; i++) {
+ body.run();
+ }
+ double[] samples = new double[iterations];
+ for (int i = 0; i < iterations; i++) {
+ long start = System.nanoTime();
+ body.run();
+ samples[i] = (System.nanoTime() - start) / 1e6;
+ }
+ double[] sorted = samples.clone();
+ Arrays.sort(sorted);
+ List<String> all = new ArrayList<>();
+ for (double sample : samples) {
+ all.add(String.format(Locale.ROOT, "%.1f", sample));
+ }
+ String line = String.format(Locale.ROOT,
+ "BENCH fork=%s requested=%d queues=%d op=%s%s warmups=%d n=%d "
+ + "median_ms=%.1f all_ms=%s",
+ fork, requested, queues, op, variant == null ? "" : "-" + variant,
+ warmups, iterations, sorted[(iterations - 1) / 2], String.join(",", all));
Review Comment:
Done.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSConfigBenchmarkGenerator.java:
##########
@@ -0,0 +1,543 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+
+import static org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableSet.of;
+
+/**
+ * Generates synthetic but structurally realistic capacity-scheduler
+ * configuration content for a requested queue count.
+ * It feeds {@link TestCapacitySchedulerConfigBenchmark} (scheduler load,
+ * refresh and validation timings) and {@link TestCSConfigBenchmarkGenerator}.
+ *
+ * Tree shape (mixed depth 3 and 4):
+ * <pre>
+ * root
+ * p0 (pct children, node label, one legacy managed parent leaf)
+ * p1 (weight children, AQC v2 template on sub-parent s0)
+ * p2 (pct children, unlabeled)
+ * p3 (weight children)
+ * ...
+ * each pI: 2 direct leaves + up to 5 sub-parents with up to 9 leaves each
+ * </pre>
+ *
+ * Capacity modes:
+ * - PCT_WEIGHT_MIXED (default): root children are percentage; even-indexed
+ * subtrees use percentage children, odd-indexed use weight children; the last
+ * sub-parent of a weight subtree flips its children back to percentage
+ * (legal per the legacy-mode parent/child mode table in AbstractParentQueue).
+ * - ALL_ABSOLUTE: every queue uses absolute [memory=..,vcores=..] resources,
+ * with parent minimums equal to the sum of their children. This is a separate
+ * mode because in legacy queue mode absolute resources cannot be mixed with
+ * percentage/weight anywhere in one hierarchy (absolute children require an
+ * absolute parent, and siblings must be mode-uniform).
+ *
+ * Extras sprinkled for realism: accessible-node-labels + per-label capacities
+ * on every fourth subtree (labels "blue"/"ssd"), legacy managed-parent
+ * (auto-create-child-queue) leaf templates, auto-queue-creation-v2 templates,
+ * legacy queue-mapping placement rules, ACLs, user-limit-factor,
+ * maximum-capacity and ordering policies on a subset of queues.
+ */
+public final class CSConfigBenchmarkGenerator {
+
+ public enum CapacityMode {
+ PCT_WEIGHT_MIXED,
+ ALL_ABSOLUTE
+ }
+
+ /** Everything the harness needs to know about a generated config. */
+ public static final class GeneratedConfig {
+ private final CapacitySchedulerConfiguration conf;
+ private final int queueCount;
+ private final List<String> leafPaths;
+ private final String mutationLeafA;
+ private final String mutationLeafB;
+ private final List<String> labels;
+
+ private GeneratedConfig(CapacitySchedulerConfiguration conf, int queueCount,
+ List<String> leafPaths, String mutationLeafA, String mutationLeafB,
+ List<String> labels) {
+ this.conf = conf;
+ this.queueCount = queueCount;
+ this.leafPaths = leafPaths;
+ this.mutationLeafA = mutationLeafA;
+ this.mutationLeafB = mutationLeafB;
+ this.labels = labels;
+ }
+
+ public CapacitySchedulerConfiguration getConf() {
+ return conf;
+ }
+
+ /** Number of generated queues, root excluded. */
+ public int getQueueCount() {
+ return queueCount;
+ }
+
+ public List<String> getLeafPaths() {
+ return Collections.unmodifiableList(leafPaths);
+ }
+
+ /** Two sibling leaves whose capacities can be swapped for a valid change. */
+ public String getMutationLeafA() {
+ return mutationLeafA;
+ }
+
+ public String getMutationLeafB() {
+ return mutationLeafB;
+ }
+
+ public List<String> getLabels() {
+ return Collections.unmodifiableList(labels);
+ }
+ }
+
+ private static final int DIRECT_LEAVES = 2;
+ private static final int SUB_PARENTS = 5;
+ private static final int LEAVES_PER_SUB_PARENT = 9;
+ private static final int MAX_MANAGED_PARENTS = 4;
+ private static final int MAX_AQC_V2_PARENTS = 4;
+ private static final String LABEL_BLUE = "blue";
+ private static final String LABEL_SSD = "ssd";
+ private static final String CS_PREFIX = CapacitySchedulerConfiguration.PREFIX;
+
+ private CSConfigBenchmarkGenerator() {
+ }
+
+ private static final class Node {
+ private final String name;
+ private final String path;
+ private final List<Node> children = new ArrayList<>();
+ private boolean managedParent;
+
+ Node(String parentPath, String name) {
+ this.name = name;
+ this.path = parentPath + "." + name;
+ }
+
+ boolean isLeaf() {
+ return children.isEmpty() && !managedParent;
+ }
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount) {
+ return generate(targetQueueCount, CapacityMode.PCT_WEIGHT_MIXED);
+ }
+
+ public static GeneratedConfig generate(int targetQueueCount, CapacityMode mode) {
+ List<Node> tops = buildForest(targetQueueCount);
+ CapacitySchedulerConfiguration conf =
+ new CapacitySchedulerConfiguration(new Configuration(false), false);
+
+ markManagedParents(tops);
+ QueuePath rootPath = new QueuePath(CapacitySchedulerConfiguration.ROOT);
+ conf.setQueues(rootPath, names(tops));
+
+ if (mode == CapacityMode.ALL_ABSOLUTE) {
+ emitAbsolute(conf, rootPath, tops);
+ } else {
+ emitMixed(conf, tops);
+ }
+
+ List<String> leafPaths = new ArrayList<>();
+ int count = 0;
+ for (Node top : tops) {
+ count += collect(top, leafPaths);
+ }
+ String[] mutationPair = pickMutationSiblings(tops);
+ setPlacementRules(conf, leafPaths);
+
+ List<String> labels = new ArrayList<>();
+ if (mode == CapacityMode.PCT_WEIGHT_MIXED) {
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null && !labels.contains(label)) {
+ labels.add(label);
+ }
+ }
+ }
+ return new GeneratedConfig(conf, count, leafPaths, mutationPair[0],
+ mutationPair[1], labels);
+ }
+
+ /**
+ * Returns a copy of {@code base} with a valid capacity change applied: one
+ * percent is moved from mutation leaf B to mutation leaf A (and likewise for
+ * their per-label capacities), so per-parent sums stay at 100.
+ */
+ public static Configuration createMutatedCopy(Configuration base,
+ GeneratedConfig gen) {
+ Configuration copy = new Configuration(base);
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), null);
+ for (String label : gen.getLabels()) {
+ moveOnePercent(copy, gen.getMutationLeafA(), gen.getMutationLeafB(), label);
+ }
+ return copy;
+ }
+
+ private static void moveOnePercent(Configuration conf, String leafA,
+ String leafB, String label) {
+ if (leafA == null || leafB == null) {
+ return;
+ }
+ String suffix = (label == null) ? ".capacity"
+ : ".accessible-node-labels." + label + ".capacity";
+ String keyA = CS_PREFIX + leafA + suffix;
+ String keyB = CS_PREFIX + leafB + suffix;
+ String a = conf.get(keyA);
+ String b = conf.get(keyB);
+ if (a == null || b == null) {
+ return;
+ }
+ try {
+ conf.set(keyA, String.valueOf(Float.parseFloat(a) + 1.0f));
+ conf.set(keyB, String.valueOf(Float.parseFloat(b) - 1.0f));
+ } catch (NumberFormatException e) {
+ // Absolute/weight profiles mutate a real queue limit, never a no-op.
+ conf.set(CS_PREFIX + leafA + ".maximum-applications", "2345");
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Tree construction
+ // ---------------------------------------------------------------------
+
+ private static List<Node> buildForest(int targetQueueCount) {
+ List<Node> tops = new ArrayList<>();
+ int fullSubtree = 1 + DIRECT_LEAVES + SUB_PARENTS * (1 + LEAVES_PER_SUB_PARENT);
+ int remaining = targetQueueCount;
+ int i = 0;
+ while (remaining > 0) {
+ int budget = Math.min(fullSubtree, remaining);
+ Node top = new Node(CapacitySchedulerConfiguration.ROOT, "p" + i);
+ budget--;
+ int directLeaves = Math.min(DIRECT_LEAVES, budget);
+ for (int l = 0; l < directLeaves; l++) {
+ top.children.add(new Node(top.path, "l" + l));
+ }
+ budget -= directLeaves;
+ int s = 0;
+ while (budget > 0) {
+ Node sub = new Node(top.path, "s" + s);
+ top.children.add(sub);
+ budget--;
+ int subLeaves = Math.min(LEAVES_PER_SUB_PARENT, budget);
+ for (int l = 0; l < subLeaves; l++) {
+ sub.children.add(new Node(sub.path, "l" + l));
+ }
+ budget -= subLeaves;
+ s++;
+ }
+ remaining -= subtreeSize(top);
+ tops.add(top);
+ i++;
+ }
+ return tops;
+ }
+
+ private static int subtreeSize(Node node) {
+ int size = 1;
+ for (Node child : node.children) {
+ size += subtreeSize(child);
+ }
+ return size;
+ }
+
+ /**
+ * Converts the last direct leaf of every other percentage subtree into a
+ * legacy managed parent (auto-create-child-queue.enabled), capped globally.
+ */
+ private static void markManagedParents(List<Node> tops) {
+ int marked = 0;
+ for (int i = 0; i < tops.size() && marked < MAX_MANAGED_PARENTS; i += 4) {
+ Node top = tops.get(i);
+ for (int c = top.children.size() - 1; c >= 0; c--) {
+ Node child = top.children.get(c);
+ if (child.isLeaf()) {
+ child.managedParent = true;
+ marked++;
+ break;
+ }
+ }
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Mixed percentage/weight emission
+ // ---------------------------------------------------------------------
+
+ private static void emitMixed(CapacitySchedulerConfiguration conf, List<Node> tops) {
+ float[] topCaps = splitPercentages(tops.size());
+ Map<String, List<Integer>> labelMembers = new LinkedHashMap<>();
+ for (int i = 0; i < tops.size(); i++) {
+ String label = labelFor(i);
+ if (label != null) {
+ labelMembers.computeIfAbsent(label, k -> new ArrayList<>()).add(i);
+ }
+ }
+ // Per-label percentage shares among the labeled root children; the sums per
+ // label are 100 among label-accessible siblings, as legacy mode requires.
+ Map<Integer, Float> topLabelCap = new LinkedHashMap<>();
+ for (List<Integer> members : labelMembers.values()) {
+ float[] shares = splitPercentages(members.size());
+ for (int m = 0; m < members.size(); m++) {
+ topLabelCap.put(members.get(m), shares[m]);
+ }
+ }
+
+ int aqcV2Marked = 0;
+ for (int i = 0; i < tops.size(); i++) {
+ Node top = tops.get(i);
+ QueuePath topPath = new QueuePath(top.path);
+ conf.setCapacity(topPath, topCaps[i]);
+ conf.setMaximumCapacity(topPath, 100f);
+ String label = labelFor(i);
+ if (label != null) {
+ conf.setAccessibleNodeLabels(topPath, of(label));
+ conf.setCapacityByLabel(topPath, label, topLabelCap.get(i));
+ }
+ boolean weightMode = (i % 2 == 1);
+ boolean markAqcV2 = weightMode && (i % 6 == 1) && aqcV2Marked < MAX_AQC_V2_PARENTS;
+ if (markAqcV2) {
+ aqcV2Marked++;
+ }
+ emitChildren(conf, top, weightMode, label, true, markAqcV2);
+ }
+ }
+
+ /**
+ * Emits capacities (and per-label capacities) for the children of
+ * {@code parent}, then recurses one level into sub-parents.
+ */
+ private static void emitChildren(CapacitySchedulerConfiguration conf, Node parent,
+ boolean weightMode, String label, boolean topLevel, boolean markAqcV2) {
+ if (parent.children.isEmpty()) {
+ return;
+ }
+ conf.setQueues(new QueuePath(parent.path), names(parent.children));
+ float[] caps = splitPercentages(parent.children.size());
+ for (int c = 0; c < parent.children.size(); c++) {
+ Node child = parent.children.get(c);
+ QueuePath childPath = new QueuePath(child.path);
+ // The last sub-parent of a top-level weight subtree flips its children
+ // back to percentage mode; mode must stay uniform among siblings only.
+ boolean childrenWeightMode = weightMode
+ && !(topLevel && c == parent.children.size() - 1 && !child.children.isEmpty());
+ if (weightMode) {
+ conf.setNonLabeledQueueWeight(childPath, (c % 4) + 1f);
+ } else {
+ conf.setCapacity(childPath, caps[c]);
+ conf.setMaximumCapacity(childPath, 100f);
+ }
+ if (label != null) {
+ conf.setAccessibleNodeLabels(childPath, of(label));
+ conf.setCapacityByLabel(childPath, label, caps[c]);
+ }
+ decorate(conf, child, c);
+ if (child.managedParent) {
+ emitManagedParent(conf, childPath, label);
+ } else if (!child.children.isEmpty()) {
+ if (markAqcV2 && c == DIRECT_LEAVES) {
+ emitAqcV2Template(conf, childPath);
+ }
+ emitChildren(conf, child, childrenWeightMode, label, false, false);
+ }
+ }
+ }
+
+ private static void emitManagedParent(CapacitySchedulerConfiguration conf,
+ QueuePath path, String label) {
+ conf.setAutoCreateChildQueueEnabled(path, true);
+ conf.setAutoCreatedLeafQueueConfigCapacity(path, 25f);
+ conf.setAutoCreatedLeafQueueConfigMaxCapacity(path, 100f);
+ if (label != null) {
+ conf.setAutoCreatedLeafQueueTemplateCapacityByLabel(path, label, 25f);
+ }
+ }
+
+ private static void emitAqcV2Template(CapacitySchedulerConfiguration conf,
+ QueuePath path) {
+ conf.setAutoQueueCreationV2Enabled(path, true);
+ conf.set(CS_PREFIX + path.getFullPath()
+ + ".auto-queue-creation-v2.template.capacity", "2w");
+ conf.set(CS_PREFIX + path.getFullPath()
+ + ".auto-queue-creation-v2.leaf-template.maximum-applications", "100");
+ }
+
+ /** Realistic per-queue extras on a deterministic subset of queues. */
+ private static void decorate(CapacitySchedulerConfiguration conf, Node node,
+ int childIndex) {
+ QueuePath path = new QueuePath(node.path);
+ if (childIndex % 3 == 0) {
+ conf.setUserLimitFactor(path, 2.0f);
+ }
+ if (childIndex % 5 == 1) {
+ conf.setAcl(path, QueueACL.SUBMIT_APPLICATIONS, "benchuser benchgroup");
+ conf.setAcl(path, QueueACL.ADMINISTER_QUEUE, "benchadmin");
+ }
+ if (node.isLeaf() && childIndex % 4 == 2) {
+ conf.setOrderingPolicy(path, CapacitySchedulerConfiguration.FAIR_APP_ORDERING_POLICY);
+ }
+ }
+
+ // ---------------------------------------------------------------------
+ // Absolute-mode emission
+ // ---------------------------------------------------------------------
+
+ private static void emitAbsolute(CapacitySchedulerConfiguration conf,
+ QueuePath rootPath, List<Node> tops) {
+ // Root keeps its fixed 100 percent capacity; setting an absolute capacity
+ // on root is rejected by CapacitySchedulerConfiguration.
+ for (Node top : tops) {
+ emitAbsoluteSubtree(conf, top);
+ }
+ }
+
+ /** Returns {memoryMb, vcores} configured for the subtree minimum. */
+ private static long[] emitAbsoluteSubtree(CapacitySchedulerConfiguration conf,
+ Node node) {
+ QueuePath path = new QueuePath(node.path);
+ if (node.children.isEmpty()) {
+ conf.setCapacity(path, "[memory=1024,vcores=1]");
+ conf.set(CS_PREFIX + node.path + ".maximum-capacity",
+ "[memory=2048,vcores=2]");
+ if (node.managedParent) {
+ conf.setAutoCreateChildQueueEnabled(path, true);
+ }
+ return new long[]{1024L, 1L};
+ }
+ conf.setQueues(path, names(node.children));
+ long mem = 0;
+ long vcores = 0;
+ for (Node child : node.children) {
+ long[] used = emitAbsoluteSubtree(conf, child);
+ mem += used[0];
+ vcores += used[1];
+ }
+ conf.setCapacity(path, "[memory=" + mem + ",vcores=" + vcores + "]");
+ conf.set(CS_PREFIX + node.path + ".maximum-capacity",
+ "[memory=" + (2 * mem) + ",vcores=" + (2 * vcores) + "]");
+ return new long[]{mem, vcores};
+ }
+
+ // ---------------------------------------------------------------------
+ // Placement rules and helpers
+ // ---------------------------------------------------------------------
+
+ private static void setPlacementRules(CapacitySchedulerConfiguration conf,
+ List<String> leafPaths) {
+ if (leafPaths.isEmpty()) {
+ return;
+ }
+ List<String> mappings = new ArrayList<>();
+ mappings.add("u:benchuser1:" + leafPaths.get(0));
+ mappings.add("u:benchuser2:" + leafPaths.get(leafPaths.size() / 2));
+ mappings.add("g:benchgroup:" + leafPaths.get(leafPaths.size() - 1));
+ if (leafPaths.size() > 3) {
+ mappings.add("u:%user:" + leafPaths.get(1));
+ }
+ conf.set(CapacitySchedulerConfiguration.QUEUE_MAPPING,
+ String.join(",", mappings));
+ }
+
+ private static int collect(Node node, List<String> leafPaths) {
+ int count = 1;
+ if (node.isLeaf()) {
+ leafPaths.add(node.path);
+ }
+ for (Node child : node.children) {
+ count += collect(child, leafPaths);
+ }
+ return count;
+ }
+
+ /**
+ * Picks two plain sibling leaves whose capacity keys can be swapped to
+ * produce a valid changed configuration. Prefers an unlabeled percentage
+ * subtree; falls back to any sub-parent, then to direct leaves.
+ */
+ private static String[] pickMutationSiblings(List<Node> tops) {
+ for (int pass = 0; pass < 2; pass++) {
+ for (int i = 0; i < tops.size(); i += 2) {
+ if (pass == 0 && labelFor(i) != null) {
+ continue;
+ }
+ Node top = tops.get(i);
+ for (Node child : top.children) {
+ List<Node> leaves = new ArrayList<>();
+ for (Node leaf : child.children) {
+ if (leaf.isLeaf()) {
+ leaves.add(leaf);
+ }
+ }
+ if (leaves.size() >= 2) {
+ return new String[]{leaves.get(0).path, leaves.get(1).path};
+ }
+ }
+ List<Node> direct = new ArrayList<>();
+ for (Node child : top.children) {
+ if (child.isLeaf()) {
+ direct.add(child);
+ }
+ }
+ if (direct.size() >= 2) {
+ return new String[]{direct.get(0).path, direct.get(1).path};
+ }
+ }
+ }
+ return new String[]{null, null};
+ }
+
+ private static String labelFor(int topIndex) {
+ if (topIndex % 4 != 0) {
+ return null;
+ }
+ return (topIndex % 8 == 0) ? LABEL_BLUE : LABEL_SSD;
+ }
+
+ private static String[] names(List<Node> nodes) {
+ String[] names = new String[nodes.size()];
+ for (int i = 0; i < nodes.size(); i++) {
+ names[i] = nodes.get(i).name;
+ }
+ return names;
+ }
+
+ /**
+ * Splits 100 percent into n two-decimal shares summing to exactly 100.00.
+ */
+ private static float[] splitPercentages(int n) {
+ float[] out = new float[n];
+ int base = 10000 / n;
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-12011. Add Capacity Scheduler configuration tests and a load, refresh and validation benchmark [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8799:
URL: https://github.com/apache/hadoop/pull/8799#issuecomment-6098757527
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 30 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 47s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 30m 56s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 31m 29s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 0m 59s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 59s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 56s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 551 unchanged - 2 fixed = 552 total (was 553) |
| +1 :green_heart: | mvnsite | 1m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 29m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 111m 24s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 235m 14s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/3/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c4c9548bae63 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d07707b418e69e2300d97c949578348ee423dccd |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/3/testReport/ |
| Max. process+thread count | 932 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8799/3/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org