You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "susheel-gupta (via GitHub)" <gi...@apache.org> on 2023/02/09 22:22:11 UTC
[GitHub] [hadoop] susheel-gupta opened a new pull request, #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
susheel-gupta opened a new pull request, #5380:
URL: https://github.com/apache/hadoop/pull/5380
Change-Id: Icb90fc5887fc282a671c92980733aed3fd7ce85f
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1425552948
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 12s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 27s | | trunk passed |
| +1 :green_heart: | compile | 1m 10s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 2s | | trunk passed |
| -1 :x: | javadoc | 0m 58s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 44s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 28m 3s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 1m 0s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 1m 0s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 1 new + 225 unchanged - 1 fixed = 226 total (was 226) |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 0m 54s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 41s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 41 new + 268 unchanged - 22 fixed = 309 total (was 290) |
| +1 :green_heart: | mvnsite | 0m 55s | | the patch passed |
| -1 :x: | javadoc | 0m 41s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 6s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 27m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 133m 13s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 253m 42s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:[line 817] |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:[line 845] |
| Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoCreatedQueuePreemption |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestAbsoluteResourceConfiguration |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig |
| | hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestAbsoluteResourceWithAutoQueue |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5ee8f3ad8317 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aa42dc2a03e482d371561e6aef986e70d7f4b15a |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/testReport/ |
| Max. process+thread count | 921 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on a diff in pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "susheel-gupta (via GitHub)" <gi...@apache.org>.
susheel-gupta commented on code in PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#discussion_r1146496279
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java:
##########
@@ -0,0 +1,1682 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableMap;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.util.Sets;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerDynamicEditException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.security.AccessControlException;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerStatus;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+import org.apache.hadoop.yarn.api.records.QueueInfo;
+import org.apache.hadoop.yarn.api.records.QueueState;
+import org.apache.hadoop.yarn.api.records.QueueUserACLInfo;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
+import org.apache.hadoop.yarn.factories.RecordFactory;
+import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.security.AccessType;
+import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerEventType;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ActiveUsersManager;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivitiesLogger;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityDiagnosticConstant;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.AllocationState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.QueueOrderingPolicy;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ContainerAllocationProposal;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ResourceCommitRequest;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.SchedulerContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSet;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSetUtils;
+import org.apache.hadoop.yarn.util.UnitsConversionUtil;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
+import org.apache.hadoop.yarn.util.resource.Resources;
+
+import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getACLsForFlexibleAutoCreatedParentQueue;
+
+public abstract class AbstractParentQueue extends AbstractCSQueue {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractParentQueue.class);
+
+ protected final List<CSQueue> childQueues;
+ private final boolean rootQueue;
+ private volatile int numApplications;
+
+ private final RecordFactory recordFactory =
+ RecordFactoryProvider.getRecordFactory(null);
+
+ private QueueOrderingPolicy queueOrderingPolicy;
+
+ private long lastSkipQueueDebugLoggingTimestamp = -1;
+
+ private int runnableApps;
+
+ private final boolean allowZeroCapacitySum;
+
+ private AutoCreatedQueueTemplate autoCreatedQueueTemplate;
+
+ // A ratio of the queue's effective minimum resource and the summary of the configured
+ // minimum resource of its children grouped by labels and calculated for each resource names
+ // distinctively.
+ private final Map<String, Map<String, Float>> effectiveMinResourceRatio =
+ new ConcurrentHashMap<>();
+
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old)
+ throws IOException {
+ this(queueContext, queueName, parent, old, false);
+ }
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old, boolean isDynamic) throws
+ IOException {
+
+ super(queueContext, queueName, parent, old);
+ setDynamicQueue(isDynamic);
+ this.rootQueue = (parent == null);
+
+ float rawCapacity = queueContext.getConfiguration()
+ .getNonLabeledQueueCapacity(this.queuePath);
+
+ if (rootQueue &&
+ (rawCapacity != CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE)) {
+ throw new IllegalArgumentException("Illegal " +
+ "capacity of " + rawCapacity + " for queue " + queueName +
+ ". Must be " + CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE);
+ }
+
+ this.childQueues = new ArrayList<>();
+ this.allowZeroCapacitySum =
+ queueContext.getConfiguration()
+ .getAllowZeroCapacitySum(getQueuePath());
+
+ }
+
+ // returns what is configured queue ordering policy
+ private String getQueueOrderingPolicyConfigName() {
+ return queueOrderingPolicy == null ?
+ null :
+ queueOrderingPolicy.getConfigName();
+ }
+
+ protected void setupQueueConfigs(Resource clusterResource)
+ throws IOException {
+ writeLock.lock();
+ try {
+ CapacitySchedulerConfiguration configuration = queueContext.getConfiguration();
+ autoCreatedQueueTemplate = new AutoCreatedQueueTemplate(
+ configuration, this.queuePath);
+ super.setupQueueConfigs(clusterResource);
+ StringBuilder aclsString = new StringBuilder();
+ for (Map.Entry<AccessType, AccessControlList> e : getACLs().entrySet()) {
+ aclsString.append(e.getKey()).append(":")
+ .append(e.getValue().getAclString());
+ }
+
+ StringBuilder labelStrBuilder = new StringBuilder();
+ if (getAccessibleNodeLabels() != null) {
+ for (String nodeLabel : getAccessibleNodeLabels()) {
+ labelStrBuilder.append(nodeLabel).append(",");
+ }
+ }
+
+ // Initialize queue ordering policy
+ queueOrderingPolicy = configuration.getQueueOrderingPolicy(
+ getQueuePath(), parent == null ?
+ null :
+ ((AbstractParentQueue) parent).getQueueOrderingPolicyConfigName());
+ queueOrderingPolicy.setQueues(childQueues);
+
+ LOG.info(getQueueName() + ", " + getCapacityOrWeightString()
+ + ", absoluteCapacity=" + getAbsoluteCapacity()
+ + ", maxCapacity=" + getMaximumCapacity()
+ + ", absoluteMaxCapacity=" + getAbsoluteMaximumCapacity()
+ + ", state=" + getState() + ", acls="
+ + aclsString + ", labels=" + labelStrBuilder + "\n"
+ + ", reservationsContinueLooking=" + isReservationsContinueLooking()
+ + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
+ + ", priority=" + getPriority()
+ + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ protected void setDynamicQueueACLProperties() {
+ super.setDynamicQueueACLProperties();
+
+ if (parent instanceof AbstractParentQueue) {
+ acls.putAll(getACLsForFlexibleAutoCreatedParentQueue(
+ ((AbstractParentQueue) parent).getAutoCreatedQueueTemplate()));
+ }
+ }
+
+ private static float PRECISION = 0.0005f; // 0.05% precision
+
+ // Check weight configuration, throw exception when configuration is invalid
+ // return true when all children use weight mode.
+ private QueueCapacityType getCapacityConfigurationTypeForQueues(
+ Collection<CSQueue> queues) throws IOException {
+ // Do we have ANY queue set capacity in any labels?
+ boolean percentageIsSet = false;
+
+ // Do we have ANY queue set weight in any labels?
+ boolean weightIsSet = false;
+
+ // Do we have ANY queue set absolute in any labels?
+ boolean absoluteMinResSet = false;
+
+ StringBuilder diagMsg = new StringBuilder();
+
+ for (CSQueue queue : queues) {
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ float capacityByLabel = queue.getQueueCapacities().getCapacity(nodeLabel);
+ if (capacityByLabel > 0) {
+ percentageIsSet = true;
+ }
+ float weightByLabel = queue.getQueueCapacities().getWeight(nodeLabel);
+ // By default weight is set to -1, so >= 0 is enough.
+ if (weightByLabel >= 0) {
+ weightIsSet = true;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses weight mode}. ");
+ }
+ if (!queue.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel)
+ .equals(Resources.none())) {
+ absoluteMinResSet = true;
+ // There's a special handling: when absolute resource is configured,
+ // capacity will be calculated (and set) for UI/metrics purposes, so
+ // when asboluteMinResource is set, unset percentage
+ percentageIsSet = false;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses absolute mode}. ");
+ }
+ if (percentageIsSet) {
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses percentage mode}. ");
+ }
+ }
+ }
+ // If we have mixed capacity, weight or absolute resource (any of the two)
+ // We will throw exception
+ // Root queue is an exception here, because by default root queue returns
+ // 100 as capacity no matter what. We should look into this case in the
+ // future. To avoid impact too many code paths, we don;t check root queue's
+ // config.
+ if (queues.iterator().hasNext() &&
+ !queues.iterator().next().getQueuePath().equals(
+ CapacitySchedulerConfiguration.ROOT) &&
+ (percentageIsSet ? 1 : 0) + (weightIsSet ? 1 : 0) + (absoluteMinResSet ?
+ 1 :
+ 0) > 1) {
+ throw new IOException("Parent queue '" + getQueuePath()
+ + "' have children queue used mixed of "
+ + " weight mode, percentage and absolute mode, it is not allowed, please "
+ + "double check, details:" + diagMsg.toString());
+ }
+
+ if (weightIsSet || queues.isEmpty()) {
+ return QueueCapacityType.WEIGHT;
+ } else if (absoluteMinResSet) {
+ return QueueCapacityType.ABSOLUTE_RESOURCE;
+ } else {
+ return QueueCapacityType.PERCENT;
+ }
+ }
+
+ private enum QueueCapacityType {
+ WEIGHT, ABSOLUTE_RESOURCE, PERCENT;
+ }
+
+ /**
+ * Set child queue and verify capacities
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | | parent-weight | parent-pct | parent-abs |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-weight | No specific check | No specific check | X |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-pct | Sum(children.capacity) = | When: | X |
+ * | | 0 OR 100 | parent.capacity>0 | |
+ * | | | sum(children.capacity)=100 OR 0 | |
+ * | | | parent.capacity=0 | |
+ * | | | sum(children.capacity)=0 | |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-abs | X | X | Sum(children.minRes)<= |
+ * | | | | parent.minRes |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * @param childQueues
+ */
+ void setChildQueues(Collection<CSQueue> childQueues) throws IOException {
+ writeLock.lock();
+ try {
+ boolean isLegacyQueueMode = queueContext.getConfiguration().isLegacyQueueMode();
+ if (isLegacyQueueMode) {
+ QueueCapacityType childrenCapacityType =
+ getCapacityConfigurationTypeForQueues(childQueues);
+ QueueCapacityType parentCapacityType =
+ getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
+
+ if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
+ || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
+ // We don't allow any mixed absolute + {weight, percentage} between
+ // children and parent
+ if (childrenCapacityType != parentCapacityType && !this.getQueuePath()
+ .equals(CapacitySchedulerConfiguration.ROOT)) {
+ throw new IOException("Parent=" + this.getQueuePath()
+ + ": When absolute minResource is used, we must make sure both "
+ + "parent and child all use absolute minResource");
+ }
+
+ // Ensure that for each parent queue: parent.min-resource >=
+ // Σ(child.min-resource).
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ Resource minRes = Resources.createResource(0, 0);
+ for (CSQueue queue : childQueues) {
+ // Accumulate all min/max resource configured for all child queues.
+ Resources.addTo(minRes, queue.getQueueResourceQuotas()
+ .getConfiguredMinResource(nodeLabel));
+ }
+ Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel,
+ queueContext.getClusterResource());
+ Resource parentMinResource =
+ usageTracker.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel);
+ if (!parentMinResource.equals(Resources.none()) && Resources.lessThan(
+ resourceCalculator, resourceByLabel, parentMinResource, minRes)) {
+ throw new IOException(
+ "Parent Queues" + " capacity: " + parentMinResource
+ + " is less than" + " to its children:" + minRes
+ + " for queue:" + getQueueName());
+ }
+ }
+ }
+
+ // When child uses percent
+ if (childrenCapacityType == QueueCapacityType.PERCENT) {
+ float childrenPctSum = 0;
+ // check label capacities
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ // check children's labels
+ childrenPctSum = 0;
+ for (CSQueue queue : childQueues) {
+ childrenPctSum += queue.getQueueCapacities().getCapacity(nodeLabel);
+ }
+
+ if (Math.abs(1 - childrenPctSum) > PRECISION) {
+ // When children's percent sum != 100%
+ if (Math.abs(childrenPctSum) > PRECISION) {
+ // It is wrong when percent sum != {0, 1}
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". It should be either 0 or 1.0");
+ } else {
+ // We also allow children's percent sum = 0 under the following
+ // conditions
+ // - Parent uses weight mode
+ // - Parent uses percent mode, and parent has
+ // (capacity=0 OR allowZero)
+ if (parentCapacityType == QueueCapacityType.PERCENT) {
+ if ((Math.abs(queueCapacities.getCapacity(nodeLabel))
+ > PRECISION) && (!allowZeroCapacitySum)) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName()
+ + " for label=" + nodeLabel
+ + ". It is set to 0, but parent percent != 0, and "
+ + "doesn't allow children capacity to set to 0");
+ }
+ }
+ }
+ } else {
+ // Even if child pct sum == 1.0, we will make sure parent has
+ // positive percent.
+ if (parentCapacityType == QueueCapacityType.PERCENT && Math.abs(
+ queueCapacities.getCapacity(nodeLabel)) <= 0f
+ && !allowZeroCapacitySum) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". queue=" + getQueueName()
+ + " has zero capacity, but child"
+ + "queues have positive capacities");
+ }
+ }
+ }
+ }
+ }
+
+ this.childQueues.clear();
+ this.childQueues.addAll(childQueues);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("setChildQueues: " + getChildQueuesToPrint());
+ }
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+
+ @Override
+ public QueueInfo getQueueInfo(
+ boolean includeChildQueues, boolean recursive) {
+ readLock.lock();
+ try {
+ QueueInfo queueInfo = getQueueInfo();
+
+ List<QueueInfo> childQueuesInfo = new ArrayList<>();
+ if (includeChildQueues) {
+ for (CSQueue child : childQueues) {
+ // Get queue information recursively?
+ childQueuesInfo.add(child.getQueueInfo(recursive, recursive));
+ }
+ }
+ queueInfo.setChildQueues(childQueuesInfo);
+
+ return queueInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+
+ private QueueUserACLInfo getUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ QueueUserACLInfo userAclInfo = recordFactory.newRecordInstance(
+ QueueUserACLInfo.class);
+ List<QueueACL> operations = new ArrayList<QueueACL>();
+ for (QueueACL operation : QueueACL.values()) {
+ if (hasAccess(operation, user)) {
+ operations.add(operation);
+ }
+ }
+
+ userAclInfo.setQueueName(getQueuePath());
+ userAclInfo.setUserAcls(operations);
+ return userAclInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ @Override
+ public List<QueueUserACLInfo> getQueueUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ List<QueueUserACLInfo> userAcls = new ArrayList<>();
+
+ // Add parent queue acls
+ userAcls.add(getUserAclInfo(user));
+
+ // Add children queue acls
+ for (CSQueue child : childQueues) {
+ userAcls.addAll(child.getQueueUserAclInfo(user));
+ }
+
+ return userAcls;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ public String toString() {
+ return getQueueName() + ": " +
+ "numChildQueue= " + childQueues.size() + ", " +
+ getCapacityOrWeightString() + ", " +
+ "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
+ "usedResources=" + usageTracker.getQueueUsage().getUsed() + ", " +
+ "usedCapacity=" + getUsedCapacity() + ", " +
+ "numApps=" + getNumApplications() + ", " +
+ "numContainers=" + getNumContainers();
+ }
+
+ private CSQueue createNewQueue(String childQueuePath, boolean isLeaf)
+ throws SchedulerDynamicEditException {
+ try {
+ AbstractCSQueue childQueue;
+ String queueShortName = childQueuePath.substring(
+ childQueuePath.lastIndexOf(".") + 1);
+
+ if (isLeaf) {
+ childQueue = new LeafQueue(queueContext,
+ queueShortName, this, null, true);
+ } else {
+ childQueue = new ParentQueue(queueContext, queueShortName, this, null, true);
+ }
+ childQueue.setDynamicQueue(true);
+ // It should be sufficient now, we don't need to set more, because weights
+ // related setup will be handled in updateClusterResources
+
+ return childQueue;
+ } catch (IOException e) {
+ throw new SchedulerDynamicEditException(e.toString());
+ }
+ }
+
+
+ public AbstractParentQueue addDynamicParentQueue(String queuePath)
Review Comment:
@p-szucs I agree with the comment you made, it would be better keep the functionality more accurate and clear.
What do you suggest @bteke ?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1443140666
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 21s | | https://github.com/apache/hadoop/pull/5380 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/3/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1482426456
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 22s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 40s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javac | 1m 7s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/5/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 generated 1 new + 225 unchanged - 1 fixed = 226 total (was 226) |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | javac | 0m 55s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/5/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 42s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 20 new + 263 unchanged - 22 fixed = 283 total (was 285) |
| +1 :green_heart: | mvnsite | 0m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | spotbugs | 2m 11s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/5/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 20m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 102m 49s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 207m 2s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:[line 744] |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:[line 772] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 245ed64b0635 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9ef7fc9469c718e6a2915227bdfafbe5428089c8 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/5/testReport/ |
| Max. process+thread count | 862 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1535613449
Thanks @susheel-gupta for the explanation,
Makes sense.
Latest patch LGTM, committed to trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth closed pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth closed pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
URL: https://github.com/apache/hadoop/pull/5380
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "susheel-gupta (via GitHub)" <gi...@apache.org>.
susheel-gupta commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1531437238
> Hi @susheel-gupta , Just took a look at your changes in this PR.
>
> I compared the old version of ParentQueue vs. AbstractParentQueue and found some differences:
>
> * the private int field with name 'numApplications' was volatile, now it is changed to AtomicInteger. What is the reason of this?
> * In the method called 'addApplication', the writelock was locked during incrementing numApplications, now I can see it removed. Do you know why this was necessary?
> * Same for method called 'removeApplication'
>
> Otherwise LGTM
@szilard-nemeth There was this [warning](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) where they asked to fix the spotbugs, so I added this [commit](https://github.com/apache/hadoop/pull/5380/commits/0f0d8f21823c23d49baa620a62111e7c3a269193) .
With the above changes in this [commit](https://github.com/apache/hadoop/pull/5380/commits/0f0d8f21823c23d49baa620a62111e7c3a269193) , the numApplications field is an instance of AtomicInteger, which provides an atomic incrementAndGet method to increment the value of the counter atomically. Since AtomicInteger provides atomic operations, we don't need to use locks or synchronization to ensure atomicity.
With these changes, the "VO_VOLATILE_INCREMENT: An increment to a volatile field isn't atomic" issue should be resolved, and the code should now increment the numApplications count atomically.
References:
- https://www.digitalocean.com/community/tutorials/atomicinteger-java
- https://www.baeldung.com/java-atomic-variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#discussion_r1138836995
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java:
##########
@@ -0,0 +1,1682 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableMap;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.util.Sets;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerDynamicEditException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.security.AccessControlException;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerStatus;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+import org.apache.hadoop.yarn.api.records.QueueInfo;
+import org.apache.hadoop.yarn.api.records.QueueState;
+import org.apache.hadoop.yarn.api.records.QueueUserACLInfo;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
+import org.apache.hadoop.yarn.factories.RecordFactory;
+import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.security.AccessType;
+import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerEventType;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ActiveUsersManager;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivitiesLogger;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityDiagnosticConstant;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.AllocationState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.QueueOrderingPolicy;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ContainerAllocationProposal;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ResourceCommitRequest;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.SchedulerContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSet;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSetUtils;
+import org.apache.hadoop.yarn.util.UnitsConversionUtil;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
+import org.apache.hadoop.yarn.util.resource.Resources;
+
+import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getACLsForFlexibleAutoCreatedParentQueue;
+
+public abstract class AbstractParentQueue extends AbstractCSQueue {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractParentQueue.class);
+
+ protected final List<CSQueue> childQueues;
+ private final boolean rootQueue;
+ private volatile int numApplications;
+
+ private final RecordFactory recordFactory =
+ RecordFactoryProvider.getRecordFactory(null);
+
+ private QueueOrderingPolicy queueOrderingPolicy;
+
+ private long lastSkipQueueDebugLoggingTimestamp = -1;
+
+ private int runnableApps;
+
+ private final boolean allowZeroCapacitySum;
+
+ private AutoCreatedQueueTemplate autoCreatedQueueTemplate;
+
+ // A ratio of the queue's effective minimum resource and the summary of the configured
+ // minimum resource of its children grouped by labels and calculated for each resource names
+ // distinctively.
+ private final Map<String, Map<String, Float>> effectiveMinResourceRatio =
+ new ConcurrentHashMap<>();
+
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old)
+ throws IOException {
+ this(queueContext, queueName, parent, old, false);
+ }
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old, boolean isDynamic) throws
+ IOException {
+
+ super(queueContext, queueName, parent, old);
+ setDynamicQueue(isDynamic);
+ this.rootQueue = (parent == null);
+
+ float rawCapacity = queueContext.getConfiguration()
+ .getNonLabeledQueueCapacity(this.queuePath);
+
+ if (rootQueue &&
+ (rawCapacity != CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE)) {
+ throw new IllegalArgumentException("Illegal " +
+ "capacity of " + rawCapacity + " for queue " + queueName +
+ ". Must be " + CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE);
+ }
+
+ this.childQueues = new ArrayList<>();
+ this.allowZeroCapacitySum =
+ queueContext.getConfiguration()
+ .getAllowZeroCapacitySum(getQueuePath());
+
+ }
+
+ // returns what is configured queue ordering policy
+ private String getQueueOrderingPolicyConfigName() {
+ return queueOrderingPolicy == null ?
+ null :
+ queueOrderingPolicy.getConfigName();
+ }
+
+ protected void setupQueueConfigs(Resource clusterResource)
+ throws IOException {
+ writeLock.lock();
+ try {
+ CapacitySchedulerConfiguration configuration = queueContext.getConfiguration();
+ autoCreatedQueueTemplate = new AutoCreatedQueueTemplate(
+ configuration, this.queuePath);
+ super.setupQueueConfigs(clusterResource);
+ StringBuilder aclsString = new StringBuilder();
+ for (Map.Entry<AccessType, AccessControlList> e : getACLs().entrySet()) {
+ aclsString.append(e.getKey()).append(":")
+ .append(e.getValue().getAclString());
+ }
+
+ StringBuilder labelStrBuilder = new StringBuilder();
+ if (getAccessibleNodeLabels() != null) {
+ for (String nodeLabel : getAccessibleNodeLabels()) {
+ labelStrBuilder.append(nodeLabel).append(",");
+ }
+ }
+
+ // Initialize queue ordering policy
+ queueOrderingPolicy = configuration.getQueueOrderingPolicy(
+ getQueuePath(), parent == null ?
+ null :
+ ((AbstractParentQueue) parent).getQueueOrderingPolicyConfigName());
+ queueOrderingPolicy.setQueues(childQueues);
+
+ LOG.info(getQueueName() + ", " + getCapacityOrWeightString()
+ + ", absoluteCapacity=" + getAbsoluteCapacity()
+ + ", maxCapacity=" + getMaximumCapacity()
+ + ", absoluteMaxCapacity=" + getAbsoluteMaximumCapacity()
+ + ", state=" + getState() + ", acls="
+ + aclsString + ", labels=" + labelStrBuilder + "\n"
+ + ", reservationsContinueLooking=" + isReservationsContinueLooking()
+ + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
+ + ", priority=" + getPriority()
+ + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ protected void setDynamicQueueACLProperties() {
+ super.setDynamicQueueACLProperties();
+
+ if (parent instanceof AbstractParentQueue) {
+ acls.putAll(getACLsForFlexibleAutoCreatedParentQueue(
+ ((AbstractParentQueue) parent).getAutoCreatedQueueTemplate()));
+ }
+ }
+
+ private static float PRECISION = 0.0005f; // 0.05% precision
+
+ // Check weight configuration, throw exception when configuration is invalid
+ // return true when all children use weight mode.
+ private QueueCapacityType getCapacityConfigurationTypeForQueues(
+ Collection<CSQueue> queues) throws IOException {
+ // Do we have ANY queue set capacity in any labels?
+ boolean percentageIsSet = false;
+
+ // Do we have ANY queue set weight in any labels?
+ boolean weightIsSet = false;
+
+ // Do we have ANY queue set absolute in any labels?
+ boolean absoluteMinResSet = false;
+
+ StringBuilder diagMsg = new StringBuilder();
+
+ for (CSQueue queue : queues) {
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ float capacityByLabel = queue.getQueueCapacities().getCapacity(nodeLabel);
+ if (capacityByLabel > 0) {
+ percentageIsSet = true;
+ }
+ float weightByLabel = queue.getQueueCapacities().getWeight(nodeLabel);
+ // By default weight is set to -1, so >= 0 is enough.
+ if (weightByLabel >= 0) {
+ weightIsSet = true;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses weight mode}. ");
+ }
+ if (!queue.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel)
+ .equals(Resources.none())) {
+ absoluteMinResSet = true;
+ // There's a special handling: when absolute resource is configured,
+ // capacity will be calculated (and set) for UI/metrics purposes, so
+ // when asboluteMinResource is set, unset percentage
+ percentageIsSet = false;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses absolute mode}. ");
+ }
+ if (percentageIsSet) {
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses percentage mode}. ");
+ }
+ }
+ }
+ // If we have mixed capacity, weight or absolute resource (any of the two)
+ // We will throw exception
+ // Root queue is an exception here, because by default root queue returns
+ // 100 as capacity no matter what. We should look into this case in the
+ // future. To avoid impact too many code paths, we don;t check root queue's
+ // config.
+ if (queues.iterator().hasNext() &&
+ !queues.iterator().next().getQueuePath().equals(
+ CapacitySchedulerConfiguration.ROOT) &&
+ (percentageIsSet ? 1 : 0) + (weightIsSet ? 1 : 0) + (absoluteMinResSet ?
+ 1 :
+ 0) > 1) {
+ throw new IOException("Parent queue '" + getQueuePath()
+ + "' have children queue used mixed of "
+ + " weight mode, percentage and absolute mode, it is not allowed, please "
+ + "double check, details:" + diagMsg.toString());
+ }
+
+ if (weightIsSet || queues.isEmpty()) {
+ return QueueCapacityType.WEIGHT;
+ } else if (absoluteMinResSet) {
+ return QueueCapacityType.ABSOLUTE_RESOURCE;
+ } else {
+ return QueueCapacityType.PERCENT;
+ }
+ }
+
+ private enum QueueCapacityType {
+ WEIGHT, ABSOLUTE_RESOURCE, PERCENT;
+ }
+
+ /**
+ * Set child queue and verify capacities
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | | parent-weight | parent-pct | parent-abs |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-weight | No specific check | No specific check | X |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-pct | Sum(children.capacity) = | When: | X |
+ * | | 0 OR 100 | parent.capacity>0 | |
+ * | | | sum(children.capacity)=100 OR 0 | |
+ * | | | parent.capacity=0 | |
+ * | | | sum(children.capacity)=0 | |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-abs | X | X | Sum(children.minRes)<= |
+ * | | | | parent.minRes |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * @param childQueues
+ */
+ void setChildQueues(Collection<CSQueue> childQueues) throws IOException {
+ writeLock.lock();
+ try {
+ boolean isLegacyQueueMode = queueContext.getConfiguration().isLegacyQueueMode();
+ if (isLegacyQueueMode) {
+ QueueCapacityType childrenCapacityType =
+ getCapacityConfigurationTypeForQueues(childQueues);
+ QueueCapacityType parentCapacityType =
+ getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
+
+ if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
+ || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
+ // We don't allow any mixed absolute + {weight, percentage} between
+ // children and parent
+ if (childrenCapacityType != parentCapacityType && !this.getQueuePath()
+ .equals(CapacitySchedulerConfiguration.ROOT)) {
+ throw new IOException("Parent=" + this.getQueuePath()
+ + ": When absolute minResource is used, we must make sure both "
+ + "parent and child all use absolute minResource");
+ }
+
+ // Ensure that for each parent queue: parent.min-resource >=
+ // Σ(child.min-resource).
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ Resource minRes = Resources.createResource(0, 0);
+ for (CSQueue queue : childQueues) {
+ // Accumulate all min/max resource configured for all child queues.
+ Resources.addTo(minRes, queue.getQueueResourceQuotas()
+ .getConfiguredMinResource(nodeLabel));
+ }
+ Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel,
+ queueContext.getClusterResource());
+ Resource parentMinResource =
+ usageTracker.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel);
+ if (!parentMinResource.equals(Resources.none()) && Resources.lessThan(
+ resourceCalculator, resourceByLabel, parentMinResource, minRes)) {
+ throw new IOException(
+ "Parent Queues" + " capacity: " + parentMinResource
+ + " is less than" + " to its children:" + minRes
+ + " for queue:" + getQueueName());
+ }
+ }
+ }
+
+ // When child uses percent
+ if (childrenCapacityType == QueueCapacityType.PERCENT) {
+ float childrenPctSum = 0;
+ // check label capacities
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ // check children's labels
+ childrenPctSum = 0;
+ for (CSQueue queue : childQueues) {
+ childrenPctSum += queue.getQueueCapacities().getCapacity(nodeLabel);
+ }
+
+ if (Math.abs(1 - childrenPctSum) > PRECISION) {
+ // When children's percent sum != 100%
+ if (Math.abs(childrenPctSum) > PRECISION) {
+ // It is wrong when percent sum != {0, 1}
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". It should be either 0 or 1.0");
+ } else {
+ // We also allow children's percent sum = 0 under the following
+ // conditions
+ // - Parent uses weight mode
+ // - Parent uses percent mode, and parent has
+ // (capacity=0 OR allowZero)
+ if (parentCapacityType == QueueCapacityType.PERCENT) {
+ if ((Math.abs(queueCapacities.getCapacity(nodeLabel))
+ > PRECISION) && (!allowZeroCapacitySum)) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName()
+ + " for label=" + nodeLabel
+ + ". It is set to 0, but parent percent != 0, and "
+ + "doesn't allow children capacity to set to 0");
+ }
+ }
+ }
+ } else {
+ // Even if child pct sum == 1.0, we will make sure parent has
+ // positive percent.
+ if (parentCapacityType == QueueCapacityType.PERCENT && Math.abs(
+ queueCapacities.getCapacity(nodeLabel)) <= 0f
+ && !allowZeroCapacitySum) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". queue=" + getQueueName()
+ + " has zero capacity, but child"
+ + "queues have positive capacities");
+ }
+ }
+ }
+ }
+ }
+
+ this.childQueues.clear();
+ this.childQueues.addAll(childQueues);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("setChildQueues: " + getChildQueuesToPrint());
+ }
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+
+ @Override
+ public QueueInfo getQueueInfo(
+ boolean includeChildQueues, boolean recursive) {
+ readLock.lock();
+ try {
+ QueueInfo queueInfo = getQueueInfo();
+
+ List<QueueInfo> childQueuesInfo = new ArrayList<>();
+ if (includeChildQueues) {
+ for (CSQueue child : childQueues) {
+ // Get queue information recursively?
+ childQueuesInfo.add(child.getQueueInfo(recursive, recursive));
+ }
+ }
+ queueInfo.setChildQueues(childQueuesInfo);
+
+ return queueInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+
+ private QueueUserACLInfo getUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ QueueUserACLInfo userAclInfo = recordFactory.newRecordInstance(
+ QueueUserACLInfo.class);
+ List<QueueACL> operations = new ArrayList<QueueACL>();
+ for (QueueACL operation : QueueACL.values()) {
+ if (hasAccess(operation, user)) {
+ operations.add(operation);
+ }
+ }
+
+ userAclInfo.setQueueName(getQueuePath());
+ userAclInfo.setUserAcls(operations);
+ return userAclInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ @Override
+ public List<QueueUserACLInfo> getQueueUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ List<QueueUserACLInfo> userAcls = new ArrayList<>();
+
+ // Add parent queue acls
+ userAcls.add(getUserAclInfo(user));
+
+ // Add children queue acls
+ for (CSQueue child : childQueues) {
+ userAcls.addAll(child.getQueueUserAclInfo(user));
+ }
+
+ return userAcls;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ public String toString() {
+ return getQueueName() + ": " +
+ "numChildQueue= " + childQueues.size() + ", " +
+ getCapacityOrWeightString() + ", " +
+ "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
+ "usedResources=" + usageTracker.getQueueUsage().getUsed() + ", " +
+ "usedCapacity=" + getUsedCapacity() + ", " +
+ "numApps=" + getNumApplications() + ", " +
+ "numContainers=" + getNumContainers();
+ }
+
+ private CSQueue createNewQueue(String childQueuePath, boolean isLeaf)
+ throws SchedulerDynamicEditException {
+ try {
+ AbstractCSQueue childQueue;
+ String queueShortName = childQueuePath.substring(
+ childQueuePath.lastIndexOf(".") + 1);
+
+ if (isLeaf) {
+ childQueue = new LeafQueue(queueContext,
+ queueShortName, this, null, true);
+ } else {
+ childQueue = new ParentQueue(queueContext, queueShortName, this, null, true);
+ }
+ childQueue.setDynamicQueue(true);
+ // It should be sufficient now, we don't need to set more, because weights
+ // related setup will be handled in updateClusterResources
+
+ return childQueue;
+ } catch (IOException e) {
+ throw new SchedulerDynamicEditException(e.toString());
+ }
+ }
+
+
+ public AbstractParentQueue addDynamicParentQueue(String queuePath)
Review Comment:
Thanks Susheel, I think it's a nice work. Regarding the new AbstractParentQueue class I think it probably would be more clear to keep here only common functionality for ParentQueue and ManagedParentQueue class. For example I think of the _addDynamicParentQueue_ and _addDynamicLeafQueue_ methods, which cannot be used by managed parents, as they cannot have parent child queues, so they only belong to ParentQueue class if I am not mistaken. I think separating the logic between these classes can make this code more straightforward. What are your thoughts @susheel-gupta, @bteke? Please correct me if I misunderstood something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1520667367
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 14s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 31s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 59s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javac | 0m 59s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/8/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 generated 1 new + 225 unchanged - 1 fixed = 226 total (was 226) |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | javac | 0m 51s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/8/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 40s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 20 new + 263 unchanged - 22 fixed = 283 total (was 285) |
| +1 :green_heart: | mvnsite | 0m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 45s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 102m 11s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 206m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7770de7f3fa6 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0f0d8f21823c23d49baa620a62111e7c3a269193 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/8/testReport/ |
| Max. process+thread count | 949 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1496252063
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 44m 44s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 55s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 37s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 53s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 1m 2s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javac | 1m 2s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 generated 1 new + 225 unchanged - 1 fixed = 226 total (was 226) |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | javac | 0m 50s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 41s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 20 new + 263 unchanged - 22 fixed = 283 total (was 285) |
| +1 :green_heart: | mvnsite | 0m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | spotbugs | 2m 6s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 25m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 102m 47s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 216m 17s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:[line 739] |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:[line 767] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux cc74d9a275c6 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d1986a75eece379315c54651fff73d61db6de8bf |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/testReport/ |
| Max. process+thread count | 939 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "szilard-nemeth (via GitHub)" <gi...@apache.org>.
szilard-nemeth commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1530426310
Hi @susheel-gupta ,
Just took a look at your changes in this PR.
I compared the old version of ParentQueue vs. AbstractParentQueue and found some differences:
- the private int field with name 'numApplications' was volatile, now it is changed to AtomicInteger. What is the reason of this?
- In the method called 'addApplication', the writelock was locked during incrementing numApplications, now I can see it removed. Do you know why this was necessary?
- Same for method called 'removeApplication'
Otherwise LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on a diff in pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "susheel-gupta (via GitHub)" <gi...@apache.org>.
susheel-gupta commented on code in PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#discussion_r1146496279
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java:
##########
@@ -0,0 +1,1682 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableMap;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.util.Sets;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerDynamicEditException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.security.AccessControlException;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerStatus;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+import org.apache.hadoop.yarn.api.records.QueueInfo;
+import org.apache.hadoop.yarn.api.records.QueueState;
+import org.apache.hadoop.yarn.api.records.QueueUserACLInfo;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
+import org.apache.hadoop.yarn.factories.RecordFactory;
+import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.security.AccessType;
+import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerEventType;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ActiveUsersManager;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivitiesLogger;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityDiagnosticConstant;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.AllocationState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.QueueOrderingPolicy;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ContainerAllocationProposal;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ResourceCommitRequest;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.SchedulerContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSet;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSetUtils;
+import org.apache.hadoop.yarn.util.UnitsConversionUtil;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
+import org.apache.hadoop.yarn.util.resource.Resources;
+
+import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getACLsForFlexibleAutoCreatedParentQueue;
+
+public abstract class AbstractParentQueue extends AbstractCSQueue {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractParentQueue.class);
+
+ protected final List<CSQueue> childQueues;
+ private final boolean rootQueue;
+ private volatile int numApplications;
+
+ private final RecordFactory recordFactory =
+ RecordFactoryProvider.getRecordFactory(null);
+
+ private QueueOrderingPolicy queueOrderingPolicy;
+
+ private long lastSkipQueueDebugLoggingTimestamp = -1;
+
+ private int runnableApps;
+
+ private final boolean allowZeroCapacitySum;
+
+ private AutoCreatedQueueTemplate autoCreatedQueueTemplate;
+
+ // A ratio of the queue's effective minimum resource and the summary of the configured
+ // minimum resource of its children grouped by labels and calculated for each resource names
+ // distinctively.
+ private final Map<String, Map<String, Float>> effectiveMinResourceRatio =
+ new ConcurrentHashMap<>();
+
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old)
+ throws IOException {
+ this(queueContext, queueName, parent, old, false);
+ }
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old, boolean isDynamic) throws
+ IOException {
+
+ super(queueContext, queueName, parent, old);
+ setDynamicQueue(isDynamic);
+ this.rootQueue = (parent == null);
+
+ float rawCapacity = queueContext.getConfiguration()
+ .getNonLabeledQueueCapacity(this.queuePath);
+
+ if (rootQueue &&
+ (rawCapacity != CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE)) {
+ throw new IllegalArgumentException("Illegal " +
+ "capacity of " + rawCapacity + " for queue " + queueName +
+ ". Must be " + CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE);
+ }
+
+ this.childQueues = new ArrayList<>();
+ this.allowZeroCapacitySum =
+ queueContext.getConfiguration()
+ .getAllowZeroCapacitySum(getQueuePath());
+
+ }
+
+ // returns what is configured queue ordering policy
+ private String getQueueOrderingPolicyConfigName() {
+ return queueOrderingPolicy == null ?
+ null :
+ queueOrderingPolicy.getConfigName();
+ }
+
+ protected void setupQueueConfigs(Resource clusterResource)
+ throws IOException {
+ writeLock.lock();
+ try {
+ CapacitySchedulerConfiguration configuration = queueContext.getConfiguration();
+ autoCreatedQueueTemplate = new AutoCreatedQueueTemplate(
+ configuration, this.queuePath);
+ super.setupQueueConfigs(clusterResource);
+ StringBuilder aclsString = new StringBuilder();
+ for (Map.Entry<AccessType, AccessControlList> e : getACLs().entrySet()) {
+ aclsString.append(e.getKey()).append(":")
+ .append(e.getValue().getAclString());
+ }
+
+ StringBuilder labelStrBuilder = new StringBuilder();
+ if (getAccessibleNodeLabels() != null) {
+ for (String nodeLabel : getAccessibleNodeLabels()) {
+ labelStrBuilder.append(nodeLabel).append(",");
+ }
+ }
+
+ // Initialize queue ordering policy
+ queueOrderingPolicy = configuration.getQueueOrderingPolicy(
+ getQueuePath(), parent == null ?
+ null :
+ ((AbstractParentQueue) parent).getQueueOrderingPolicyConfigName());
+ queueOrderingPolicy.setQueues(childQueues);
+
+ LOG.info(getQueueName() + ", " + getCapacityOrWeightString()
+ + ", absoluteCapacity=" + getAbsoluteCapacity()
+ + ", maxCapacity=" + getMaximumCapacity()
+ + ", absoluteMaxCapacity=" + getAbsoluteMaximumCapacity()
+ + ", state=" + getState() + ", acls="
+ + aclsString + ", labels=" + labelStrBuilder + "\n"
+ + ", reservationsContinueLooking=" + isReservationsContinueLooking()
+ + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
+ + ", priority=" + getPriority()
+ + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ protected void setDynamicQueueACLProperties() {
+ super.setDynamicQueueACLProperties();
+
+ if (parent instanceof AbstractParentQueue) {
+ acls.putAll(getACLsForFlexibleAutoCreatedParentQueue(
+ ((AbstractParentQueue) parent).getAutoCreatedQueueTemplate()));
+ }
+ }
+
+ private static float PRECISION = 0.0005f; // 0.05% precision
+
+ // Check weight configuration, throw exception when configuration is invalid
+ // return true when all children use weight mode.
+ private QueueCapacityType getCapacityConfigurationTypeForQueues(
+ Collection<CSQueue> queues) throws IOException {
+ // Do we have ANY queue set capacity in any labels?
+ boolean percentageIsSet = false;
+
+ // Do we have ANY queue set weight in any labels?
+ boolean weightIsSet = false;
+
+ // Do we have ANY queue set absolute in any labels?
+ boolean absoluteMinResSet = false;
+
+ StringBuilder diagMsg = new StringBuilder();
+
+ for (CSQueue queue : queues) {
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ float capacityByLabel = queue.getQueueCapacities().getCapacity(nodeLabel);
+ if (capacityByLabel > 0) {
+ percentageIsSet = true;
+ }
+ float weightByLabel = queue.getQueueCapacities().getWeight(nodeLabel);
+ // By default weight is set to -1, so >= 0 is enough.
+ if (weightByLabel >= 0) {
+ weightIsSet = true;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses weight mode}. ");
+ }
+ if (!queue.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel)
+ .equals(Resources.none())) {
+ absoluteMinResSet = true;
+ // There's a special handling: when absolute resource is configured,
+ // capacity will be calculated (and set) for UI/metrics purposes, so
+ // when asboluteMinResource is set, unset percentage
+ percentageIsSet = false;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses absolute mode}. ");
+ }
+ if (percentageIsSet) {
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses percentage mode}. ");
+ }
+ }
+ }
+ // If we have mixed capacity, weight or absolute resource (any of the two)
+ // We will throw exception
+ // Root queue is an exception here, because by default root queue returns
+ // 100 as capacity no matter what. We should look into this case in the
+ // future. To avoid impact too many code paths, we don;t check root queue's
+ // config.
+ if (queues.iterator().hasNext() &&
+ !queues.iterator().next().getQueuePath().equals(
+ CapacitySchedulerConfiguration.ROOT) &&
+ (percentageIsSet ? 1 : 0) + (weightIsSet ? 1 : 0) + (absoluteMinResSet ?
+ 1 :
+ 0) > 1) {
+ throw new IOException("Parent queue '" + getQueuePath()
+ + "' have children queue used mixed of "
+ + " weight mode, percentage and absolute mode, it is not allowed, please "
+ + "double check, details:" + diagMsg.toString());
+ }
+
+ if (weightIsSet || queues.isEmpty()) {
+ return QueueCapacityType.WEIGHT;
+ } else if (absoluteMinResSet) {
+ return QueueCapacityType.ABSOLUTE_RESOURCE;
+ } else {
+ return QueueCapacityType.PERCENT;
+ }
+ }
+
+ private enum QueueCapacityType {
+ WEIGHT, ABSOLUTE_RESOURCE, PERCENT;
+ }
+
+ /**
+ * Set child queue and verify capacities
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | | parent-weight | parent-pct | parent-abs |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-weight | No specific check | No specific check | X |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-pct | Sum(children.capacity) = | When: | X |
+ * | | 0 OR 100 | parent.capacity>0 | |
+ * | | | sum(children.capacity)=100 OR 0 | |
+ * | | | parent.capacity=0 | |
+ * | | | sum(children.capacity)=0 | |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-abs | X | X | Sum(children.minRes)<= |
+ * | | | | parent.minRes |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * @param childQueues
+ */
+ void setChildQueues(Collection<CSQueue> childQueues) throws IOException {
+ writeLock.lock();
+ try {
+ boolean isLegacyQueueMode = queueContext.getConfiguration().isLegacyQueueMode();
+ if (isLegacyQueueMode) {
+ QueueCapacityType childrenCapacityType =
+ getCapacityConfigurationTypeForQueues(childQueues);
+ QueueCapacityType parentCapacityType =
+ getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
+
+ if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
+ || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
+ // We don't allow any mixed absolute + {weight, percentage} between
+ // children and parent
+ if (childrenCapacityType != parentCapacityType && !this.getQueuePath()
+ .equals(CapacitySchedulerConfiguration.ROOT)) {
+ throw new IOException("Parent=" + this.getQueuePath()
+ + ": When absolute minResource is used, we must make sure both "
+ + "parent and child all use absolute minResource");
+ }
+
+ // Ensure that for each parent queue: parent.min-resource >=
+ // Σ(child.min-resource).
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ Resource minRes = Resources.createResource(0, 0);
+ for (CSQueue queue : childQueues) {
+ // Accumulate all min/max resource configured for all child queues.
+ Resources.addTo(minRes, queue.getQueueResourceQuotas()
+ .getConfiguredMinResource(nodeLabel));
+ }
+ Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel,
+ queueContext.getClusterResource());
+ Resource parentMinResource =
+ usageTracker.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel);
+ if (!parentMinResource.equals(Resources.none()) && Resources.lessThan(
+ resourceCalculator, resourceByLabel, parentMinResource, minRes)) {
+ throw new IOException(
+ "Parent Queues" + " capacity: " + parentMinResource
+ + " is less than" + " to its children:" + minRes
+ + " for queue:" + getQueueName());
+ }
+ }
+ }
+
+ // When child uses percent
+ if (childrenCapacityType == QueueCapacityType.PERCENT) {
+ float childrenPctSum = 0;
+ // check label capacities
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ // check children's labels
+ childrenPctSum = 0;
+ for (CSQueue queue : childQueues) {
+ childrenPctSum += queue.getQueueCapacities().getCapacity(nodeLabel);
+ }
+
+ if (Math.abs(1 - childrenPctSum) > PRECISION) {
+ // When children's percent sum != 100%
+ if (Math.abs(childrenPctSum) > PRECISION) {
+ // It is wrong when percent sum != {0, 1}
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". It should be either 0 or 1.0");
+ } else {
+ // We also allow children's percent sum = 0 under the following
+ // conditions
+ // - Parent uses weight mode
+ // - Parent uses percent mode, and parent has
+ // (capacity=0 OR allowZero)
+ if (parentCapacityType == QueueCapacityType.PERCENT) {
+ if ((Math.abs(queueCapacities.getCapacity(nodeLabel))
+ > PRECISION) && (!allowZeroCapacitySum)) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName()
+ + " for label=" + nodeLabel
+ + ". It is set to 0, but parent percent != 0, and "
+ + "doesn't allow children capacity to set to 0");
+ }
+ }
+ }
+ } else {
+ // Even if child pct sum == 1.0, we will make sure parent has
+ // positive percent.
+ if (parentCapacityType == QueueCapacityType.PERCENT && Math.abs(
+ queueCapacities.getCapacity(nodeLabel)) <= 0f
+ && !allowZeroCapacitySum) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". queue=" + getQueueName()
+ + " has zero capacity, but child"
+ + "queues have positive capacities");
+ }
+ }
+ }
+ }
+ }
+
+ this.childQueues.clear();
+ this.childQueues.addAll(childQueues);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("setChildQueues: " + getChildQueuesToPrint());
+ }
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+
+ @Override
+ public QueueInfo getQueueInfo(
+ boolean includeChildQueues, boolean recursive) {
+ readLock.lock();
+ try {
+ QueueInfo queueInfo = getQueueInfo();
+
+ List<QueueInfo> childQueuesInfo = new ArrayList<>();
+ if (includeChildQueues) {
+ for (CSQueue child : childQueues) {
+ // Get queue information recursively?
+ childQueuesInfo.add(child.getQueueInfo(recursive, recursive));
+ }
+ }
+ queueInfo.setChildQueues(childQueuesInfo);
+
+ return queueInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+
+ private QueueUserACLInfo getUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ QueueUserACLInfo userAclInfo = recordFactory.newRecordInstance(
+ QueueUserACLInfo.class);
+ List<QueueACL> operations = new ArrayList<QueueACL>();
+ for (QueueACL operation : QueueACL.values()) {
+ if (hasAccess(operation, user)) {
+ operations.add(operation);
+ }
+ }
+
+ userAclInfo.setQueueName(getQueuePath());
+ userAclInfo.setUserAcls(operations);
+ return userAclInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ @Override
+ public List<QueueUserACLInfo> getQueueUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ List<QueueUserACLInfo> userAcls = new ArrayList<>();
+
+ // Add parent queue acls
+ userAcls.add(getUserAclInfo(user));
+
+ // Add children queue acls
+ for (CSQueue child : childQueues) {
+ userAcls.addAll(child.getQueueUserAclInfo(user));
+ }
+
+ return userAcls;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ public String toString() {
+ return getQueueName() + ": " +
+ "numChildQueue= " + childQueues.size() + ", " +
+ getCapacityOrWeightString() + ", " +
+ "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
+ "usedResources=" + usageTracker.getQueueUsage().getUsed() + ", " +
+ "usedCapacity=" + getUsedCapacity() + ", " +
+ "numApps=" + getNumApplications() + ", " +
+ "numContainers=" + getNumContainers();
+ }
+
+ private CSQueue createNewQueue(String childQueuePath, boolean isLeaf)
+ throws SchedulerDynamicEditException {
+ try {
+ AbstractCSQueue childQueue;
+ String queueShortName = childQueuePath.substring(
+ childQueuePath.lastIndexOf(".") + 1);
+
+ if (isLeaf) {
+ childQueue = new LeafQueue(queueContext,
+ queueShortName, this, null, true);
+ } else {
+ childQueue = new ParentQueue(queueContext, queueShortName, this, null, true);
+ }
+ childQueue.setDynamicQueue(true);
+ // It should be sufficient now, we don't need to set more, because weights
+ // related setup will be handled in updateClusterResources
+
+ return childQueue;
+ } catch (IOException e) {
+ throw new SchedulerDynamicEditException(e.toString());
+ }
+ }
+
+
+ public AbstractParentQueue addDynamicParentQueue(String queuePath)
Review Comment:
@p-szucs I agree with the comment you made, it would be better, if we keep the functionality at more accurate place.
What do you suggest @bteke ?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#discussion_r1146499856
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java:
##########
@@ -0,0 +1,1682 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableMap;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.util.Sets;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerDynamicEditException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.security.AccessControlException;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerStatus;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.QueueACL;
+import org.apache.hadoop.yarn.api.records.QueueInfo;
+import org.apache.hadoop.yarn.api.records.QueueState;
+import org.apache.hadoop.yarn.api.records.QueueUserACLInfo;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceInformation;
+import org.apache.hadoop.yarn.factories.RecordFactory;
+import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.security.AccessType;
+import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerEventType;
+import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ActiveUsersManager;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivitiesLogger;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityDiagnosticConstant;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.ActivityState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.AllocationState;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.QueueOrderingPolicy;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ContainerAllocationProposal;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ResourceCommitRequest;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.SchedulerContainer;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSet;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.CandidateNodeSetUtils;
+import org.apache.hadoop.yarn.util.UnitsConversionUtil;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
+import org.apache.hadoop.yarn.util.resource.Resources;
+
+import static org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getACLsForFlexibleAutoCreatedParentQueue;
+
+public abstract class AbstractParentQueue extends AbstractCSQueue {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractParentQueue.class);
+
+ protected final List<CSQueue> childQueues;
+ private final boolean rootQueue;
+ private volatile int numApplications;
+
+ private final RecordFactory recordFactory =
+ RecordFactoryProvider.getRecordFactory(null);
+
+ private QueueOrderingPolicy queueOrderingPolicy;
+
+ private long lastSkipQueueDebugLoggingTimestamp = -1;
+
+ private int runnableApps;
+
+ private final boolean allowZeroCapacitySum;
+
+ private AutoCreatedQueueTemplate autoCreatedQueueTemplate;
+
+ // A ratio of the queue's effective minimum resource and the summary of the configured
+ // minimum resource of its children grouped by labels and calculated for each resource names
+ // distinctively.
+ private final Map<String, Map<String, Float>> effectiveMinResourceRatio =
+ new ConcurrentHashMap<>();
+
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old)
+ throws IOException {
+ this(queueContext, queueName, parent, old, false);
+ }
+
+ public AbstractParentQueue(CapacitySchedulerQueueContext queueContext,
+ String queueName, CSQueue parent, CSQueue old, boolean isDynamic) throws
+ IOException {
+
+ super(queueContext, queueName, parent, old);
+ setDynamicQueue(isDynamic);
+ this.rootQueue = (parent == null);
+
+ float rawCapacity = queueContext.getConfiguration()
+ .getNonLabeledQueueCapacity(this.queuePath);
+
+ if (rootQueue &&
+ (rawCapacity != CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE)) {
+ throw new IllegalArgumentException("Illegal " +
+ "capacity of " + rawCapacity + " for queue " + queueName +
+ ". Must be " + CapacitySchedulerConfiguration.MAXIMUM_CAPACITY_VALUE);
+ }
+
+ this.childQueues = new ArrayList<>();
+ this.allowZeroCapacitySum =
+ queueContext.getConfiguration()
+ .getAllowZeroCapacitySum(getQueuePath());
+
+ }
+
+ // returns what is configured queue ordering policy
+ private String getQueueOrderingPolicyConfigName() {
+ return queueOrderingPolicy == null ?
+ null :
+ queueOrderingPolicy.getConfigName();
+ }
+
+ protected void setupQueueConfigs(Resource clusterResource)
+ throws IOException {
+ writeLock.lock();
+ try {
+ CapacitySchedulerConfiguration configuration = queueContext.getConfiguration();
+ autoCreatedQueueTemplate = new AutoCreatedQueueTemplate(
+ configuration, this.queuePath);
+ super.setupQueueConfigs(clusterResource);
+ StringBuilder aclsString = new StringBuilder();
+ for (Map.Entry<AccessType, AccessControlList> e : getACLs().entrySet()) {
+ aclsString.append(e.getKey()).append(":")
+ .append(e.getValue().getAclString());
+ }
+
+ StringBuilder labelStrBuilder = new StringBuilder();
+ if (getAccessibleNodeLabels() != null) {
+ for (String nodeLabel : getAccessibleNodeLabels()) {
+ labelStrBuilder.append(nodeLabel).append(",");
+ }
+ }
+
+ // Initialize queue ordering policy
+ queueOrderingPolicy = configuration.getQueueOrderingPolicy(
+ getQueuePath(), parent == null ?
+ null :
+ ((AbstractParentQueue) parent).getQueueOrderingPolicyConfigName());
+ queueOrderingPolicy.setQueues(childQueues);
+
+ LOG.info(getQueueName() + ", " + getCapacityOrWeightString()
+ + ", absoluteCapacity=" + getAbsoluteCapacity()
+ + ", maxCapacity=" + getMaximumCapacity()
+ + ", absoluteMaxCapacity=" + getAbsoluteMaximumCapacity()
+ + ", state=" + getState() + ", acls="
+ + aclsString + ", labels=" + labelStrBuilder + "\n"
+ + ", reservationsContinueLooking=" + isReservationsContinueLooking()
+ + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
+ + ", priority=" + getPriority()
+ + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+ @Override
+ protected void setDynamicQueueACLProperties() {
+ super.setDynamicQueueACLProperties();
+
+ if (parent instanceof AbstractParentQueue) {
+ acls.putAll(getACLsForFlexibleAutoCreatedParentQueue(
+ ((AbstractParentQueue) parent).getAutoCreatedQueueTemplate()));
+ }
+ }
+
+ private static float PRECISION = 0.0005f; // 0.05% precision
+
+ // Check weight configuration, throw exception when configuration is invalid
+ // return true when all children use weight mode.
+ private QueueCapacityType getCapacityConfigurationTypeForQueues(
+ Collection<CSQueue> queues) throws IOException {
+ // Do we have ANY queue set capacity in any labels?
+ boolean percentageIsSet = false;
+
+ // Do we have ANY queue set weight in any labels?
+ boolean weightIsSet = false;
+
+ // Do we have ANY queue set absolute in any labels?
+ boolean absoluteMinResSet = false;
+
+ StringBuilder diagMsg = new StringBuilder();
+
+ for (CSQueue queue : queues) {
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ float capacityByLabel = queue.getQueueCapacities().getCapacity(nodeLabel);
+ if (capacityByLabel > 0) {
+ percentageIsSet = true;
+ }
+ float weightByLabel = queue.getQueueCapacities().getWeight(nodeLabel);
+ // By default weight is set to -1, so >= 0 is enough.
+ if (weightByLabel >= 0) {
+ weightIsSet = true;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses weight mode}. ");
+ }
+ if (!queue.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel)
+ .equals(Resources.none())) {
+ absoluteMinResSet = true;
+ // There's a special handling: when absolute resource is configured,
+ // capacity will be calculated (and set) for UI/metrics purposes, so
+ // when asboluteMinResource is set, unset percentage
+ percentageIsSet = false;
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses absolute mode}. ");
+ }
+ if (percentageIsSet) {
+ diagMsg.append(
+ "{Queue=" + queue.getQueuePath() + ", label=" + nodeLabel
+ + " uses percentage mode}. ");
+ }
+ }
+ }
+ // If we have mixed capacity, weight or absolute resource (any of the two)
+ // We will throw exception
+ // Root queue is an exception here, because by default root queue returns
+ // 100 as capacity no matter what. We should look into this case in the
+ // future. To avoid impact too many code paths, we don;t check root queue's
+ // config.
+ if (queues.iterator().hasNext() &&
+ !queues.iterator().next().getQueuePath().equals(
+ CapacitySchedulerConfiguration.ROOT) &&
+ (percentageIsSet ? 1 : 0) + (weightIsSet ? 1 : 0) + (absoluteMinResSet ?
+ 1 :
+ 0) > 1) {
+ throw new IOException("Parent queue '" + getQueuePath()
+ + "' have children queue used mixed of "
+ + " weight mode, percentage and absolute mode, it is not allowed, please "
+ + "double check, details:" + diagMsg.toString());
+ }
+
+ if (weightIsSet || queues.isEmpty()) {
+ return QueueCapacityType.WEIGHT;
+ } else if (absoluteMinResSet) {
+ return QueueCapacityType.ABSOLUTE_RESOURCE;
+ } else {
+ return QueueCapacityType.PERCENT;
+ }
+ }
+
+ private enum QueueCapacityType {
+ WEIGHT, ABSOLUTE_RESOURCE, PERCENT;
+ }
+
+ /**
+ * Set child queue and verify capacities
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | | parent-weight | parent-pct | parent-abs |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-weight | No specific check | No specific check | X |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-pct | Sum(children.capacity) = | When: | X |
+ * | | 0 OR 100 | parent.capacity>0 | |
+ * | | | sum(children.capacity)=100 OR 0 | |
+ * | | | parent.capacity=0 | |
+ * | | | sum(children.capacity)=0 | |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * | child-abs | X | X | Sum(children.minRes)<= |
+ * | | | | parent.minRes |
+ * +--------------+---------------------------+-------------------------------------+------------------------+
+ * @param childQueues
+ */
+ void setChildQueues(Collection<CSQueue> childQueues) throws IOException {
+ writeLock.lock();
+ try {
+ boolean isLegacyQueueMode = queueContext.getConfiguration().isLegacyQueueMode();
+ if (isLegacyQueueMode) {
+ QueueCapacityType childrenCapacityType =
+ getCapacityConfigurationTypeForQueues(childQueues);
+ QueueCapacityType parentCapacityType =
+ getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
+
+ if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
+ || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
+ // We don't allow any mixed absolute + {weight, percentage} between
+ // children and parent
+ if (childrenCapacityType != parentCapacityType && !this.getQueuePath()
+ .equals(CapacitySchedulerConfiguration.ROOT)) {
+ throw new IOException("Parent=" + this.getQueuePath()
+ + ": When absolute minResource is used, we must make sure both "
+ + "parent and child all use absolute minResource");
+ }
+
+ // Ensure that for each parent queue: parent.min-resource >=
+ // Σ(child.min-resource).
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ Resource minRes = Resources.createResource(0, 0);
+ for (CSQueue queue : childQueues) {
+ // Accumulate all min/max resource configured for all child queues.
+ Resources.addTo(minRes, queue.getQueueResourceQuotas()
+ .getConfiguredMinResource(nodeLabel));
+ }
+ Resource resourceByLabel = labelManager.getResourceByLabel(nodeLabel,
+ queueContext.getClusterResource());
+ Resource parentMinResource =
+ usageTracker.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel);
+ if (!parentMinResource.equals(Resources.none()) && Resources.lessThan(
+ resourceCalculator, resourceByLabel, parentMinResource, minRes)) {
+ throw new IOException(
+ "Parent Queues" + " capacity: " + parentMinResource
+ + " is less than" + " to its children:" + minRes
+ + " for queue:" + getQueueName());
+ }
+ }
+ }
+
+ // When child uses percent
+ if (childrenCapacityType == QueueCapacityType.PERCENT) {
+ float childrenPctSum = 0;
+ // check label capacities
+ for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
+ // check children's labels
+ childrenPctSum = 0;
+ for (CSQueue queue : childQueues) {
+ childrenPctSum += queue.getQueueCapacities().getCapacity(nodeLabel);
+ }
+
+ if (Math.abs(1 - childrenPctSum) > PRECISION) {
+ // When children's percent sum != 100%
+ if (Math.abs(childrenPctSum) > PRECISION) {
+ // It is wrong when percent sum != {0, 1}
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". It should be either 0 or 1.0");
+ } else {
+ // We also allow children's percent sum = 0 under the following
+ // conditions
+ // - Parent uses weight mode
+ // - Parent uses percent mode, and parent has
+ // (capacity=0 OR allowZero)
+ if (parentCapacityType == QueueCapacityType.PERCENT) {
+ if ((Math.abs(queueCapacities.getCapacity(nodeLabel))
+ > PRECISION) && (!allowZeroCapacitySum)) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName()
+ + " for label=" + nodeLabel
+ + ". It is set to 0, but parent percent != 0, and "
+ + "doesn't allow children capacity to set to 0");
+ }
+ }
+ }
+ } else {
+ // Even if child pct sum == 1.0, we will make sure parent has
+ // positive percent.
+ if (parentCapacityType == QueueCapacityType.PERCENT && Math.abs(
+ queueCapacities.getCapacity(nodeLabel)) <= 0f
+ && !allowZeroCapacitySum) {
+ throw new IOException(
+ "Illegal" + " capacity sum of " + childrenPctSum
+ + " for children of queue " + getQueueName() + " for label="
+ + nodeLabel + ". queue=" + getQueueName()
+ + " has zero capacity, but child"
+ + "queues have positive capacities");
+ }
+ }
+ }
+ }
+ }
+
+ this.childQueues.clear();
+ this.childQueues.addAll(childQueues);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("setChildQueues: " + getChildQueuesToPrint());
+ }
+ } finally {
+ writeLock.unlock();
+ }
+ }
+
+
+ @Override
+ public QueueInfo getQueueInfo(
+ boolean includeChildQueues, boolean recursive) {
+ readLock.lock();
+ try {
+ QueueInfo queueInfo = getQueueInfo();
+
+ List<QueueInfo> childQueuesInfo = new ArrayList<>();
+ if (includeChildQueues) {
+ for (CSQueue child : childQueues) {
+ // Get queue information recursively?
+ childQueuesInfo.add(child.getQueueInfo(recursive, recursive));
+ }
+ }
+ queueInfo.setChildQueues(childQueuesInfo);
+
+ return queueInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+
+ private QueueUserACLInfo getUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ QueueUserACLInfo userAclInfo = recordFactory.newRecordInstance(
+ QueueUserACLInfo.class);
+ List<QueueACL> operations = new ArrayList<QueueACL>();
+ for (QueueACL operation : QueueACL.values()) {
+ if (hasAccess(operation, user)) {
+ operations.add(operation);
+ }
+ }
+
+ userAclInfo.setQueueName(getQueuePath());
+ userAclInfo.setUserAcls(operations);
+ return userAclInfo;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ @Override
+ public List<QueueUserACLInfo> getQueueUserAclInfo(
+ UserGroupInformation user) {
+ readLock.lock();
+ try {
+ List<QueueUserACLInfo> userAcls = new ArrayList<>();
+
+ // Add parent queue acls
+ userAcls.add(getUserAclInfo(user));
+
+ // Add children queue acls
+ for (CSQueue child : childQueues) {
+ userAcls.addAll(child.getQueueUserAclInfo(user));
+ }
+
+ return userAcls;
+ } finally {
+ readLock.unlock();
+ }
+
+ }
+
+ public String toString() {
+ return getQueueName() + ": " +
+ "numChildQueue= " + childQueues.size() + ", " +
+ getCapacityOrWeightString() + ", " +
+ "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
+ "usedResources=" + usageTracker.getQueueUsage().getUsed() + ", " +
+ "usedCapacity=" + getUsedCapacity() + ", " +
+ "numApps=" + getNumApplications() + ", " +
+ "numContainers=" + getNumContainers();
+ }
+
+ private CSQueue createNewQueue(String childQueuePath, boolean isLeaf)
+ throws SchedulerDynamicEditException {
+ try {
+ AbstractCSQueue childQueue;
+ String queueShortName = childQueuePath.substring(
+ childQueuePath.lastIndexOf(".") + 1);
+
+ if (isLeaf) {
+ childQueue = new LeafQueue(queueContext,
+ queueShortName, this, null, true);
+ } else {
+ childQueue = new ParentQueue(queueContext, queueShortName, this, null, true);
+ }
+ childQueue.setDynamicQueue(true);
+ // It should be sufficient now, we don't need to set more, because weights
+ // related setup will be handled in updateClusterResources
+
+ return childQueue;
+ } catch (IOException e) {
+ throw new SchedulerDynamicEditException(e.toString());
+ }
+ }
+
+
+ public AbstractParentQueue addDynamicParentQueue(String queuePath)
Review Comment:
Completely agree @p-szucs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1441258209
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 1s | | Docker mode activated. |
| -1 :x: | patch | 0m 19s | | https://github.com/apache/hadoop/pull/5380 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/2/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1443826565
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 3s | | trunk passed |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 8s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 27s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 27m 44s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 1m 5s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/4/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 1 new + 225 unchanged - 1 fixed = 226 total (was 226) |
| +1 :green_heart: | compile | 0m 59s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 0m 59s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/4/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 42s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 41 new + 263 unchanged - 22 fixed = 304 total (was 285) |
| +1 :green_heart: | mvnsite | 1m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 28s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/4/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 28m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 111m 28s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 231m 58s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:[line 817] |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:[line 845] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e17f052259a4 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2719bd2dc5f3771d60d17186b4a8a5c686df3e4a |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/4/testReport/ |
| Max. process+thread count | 898 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5380: YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5380:
URL: https://github.com/apache/hadoop/pull/5380#issuecomment-1498887875
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 29s | | trunk passed |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 2m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 49s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 6s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javac | 1m 6s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 generated 1 new + 225 unchanged - 1 fixed = 226 total (was 226) |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | javac | 0m 54s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 43s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 20 new + 263 unchanged - 22 fixed = 283 total (was 285) |
| +1 :green_heart: | mvnsite | 0m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | spotbugs | 2m 18s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 24m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 101m 54s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 211m 58s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.addApplication(ApplicationId, String) At AbstractParentQueue.java:[line 739] |
| | Increment of volatile field org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.numApplications in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractParentQueue.removeApplication(ApplicationId, String) At AbstractParentQueue.java:[line 767] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5380 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 15407c880a50 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d207d51da9dec89e4ec65cb9d06bf981e60e0da3 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/testReport/ |
| Max. process+thread count | 854 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5380/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org