You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "susheel-gupta (via GitHub)" <gi...@apache.org> on 2023/01/23 06:44:02 UTC
[GitHub] [hadoop] susheel-gupta opened a new pull request, #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
susheel-gupta opened a new pull request, #5320:
URL: https://github.com/apache/hadoop/pull/5320
…CapacitySchedulerConfiguration object
Change-Id: Ifdab821bee6f0f6db4f8b17208d01cf3901820b7
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087785053
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -508,29 +521,29 @@ public void testReservationSystemNotSupported() {
}
private void assertNoValueForQueues(Set<String> queues, String postfix,
- Configuration config) {
+ CapacitySchedulerConfiguration capacitySchedulerConfig) {
for (String queue : queues) {
String key = PREFIX + queue + postfix;
assertNull("Key " + key + " has value, but it should be null",
- config.get(key));
+ capacitySchedulerConfig.get(key));
}
}
private void assertValueForQueues(Set<String> queues, String postfix,
- Configuration config, String expectedValue) {
+ CapacitySchedulerConfiguration capacitySchedulerConfig, String expectedValue) {
for (String queue : queues) {
String key = PREFIX + queue + postfix;
assertEquals("Key " + key + " has different value",
- expectedValue, config.get(key));
+ expectedValue, capacitySchedulerConfig.get(key));
Review Comment:
The same applies here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1632075650
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 38s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 35m 1s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 45s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 33 unchanged - 0 fixed = 39 total (was 33) |
| +1 :green_heart: | mvnsite | 0m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| -1 :x: | spotbugs | 2m 0s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/5/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 34m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 100m 37s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 236m 27s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForQueue(String) At CapacitySchedulerConfiguration.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForQueue(String) At CapacitySchedulerConfiguration.java:[line 1888] |
| | Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForUser(String) At CapacitySchedulerConfiguration.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForUser(String) At CapacitySchedulerConfiguration.java:[line 1875] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 33e4c52cf6ca 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce76e8ae7a0f0a1658f1078953f645613cccb9dd |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/5/testReport/ |
| Max. process+thread count | 998 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1093432627
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -723,6 +727,17 @@ public <S extends SchedulableEntity> OrderingPolicy<S> getAppOrderingPolicy(
return orderingPolicy;
}
+ public void setOrderingPolicy(String queue,
+ String appOrderingPolicy, String postfix, boolean value) {
+ setBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, value);
+ }
+
+ public boolean getOrderingPolicy(String queue, String appOrderingPolicy, String postfix) {
+ return getBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, false);
+ }
Review Comment:
This naming is really misleading, as it doesn't get the ordering policy, but it gets the properties of that policy. Since the ordering policy is pluggable, anyone can write one I think we shouldn't introduce getters for it, especially not fixed return type ones.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -536,6 +536,10 @@ public void setMaximumApplicationMasterResourcePerQueuePercent(String queue,
setFloat(getQueuePrefix(queue) + MAXIMUM_AM_RESOURCE_SUFFIX, percent);
}
+ public void setMaximumApplicationMasterResourcePercent(float percent) {
Review Comment:
Nit: a getter exists for this property, could you please move and rename this setter to be more in line of that method? It's called: getMaximumApplicationMasterResourcePercent
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1203,6 +1218,16 @@ public void reinitializeConfigurationProperties() {
configurationProperties = new ConfigurationProperties(props);
}
+ public void setQueueMaximumAllocationMb(String queue, String value) {
+ String queuePrefix = getQueuePrefix(queue);
+ set(queuePrefix + "maximum-allocation-mb", value);
Review Comment:
Please use the static constants instead of literals.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1693,6 +1726,14 @@ public boolean getIntraQueuePreemptionDisabled(String queue,
+ QUEUE_PREEMPTION_DISABLED, defaultVal);
}
+ public void setDisablePreemptionForPreemtionModes(String preemptionType, boolean value) {
+ setBoolean(PREEMPTION_CONFIG_PREFIX + preemptionType, value);
+ }
+
+ public boolean getDisablePreemptionForObserveOnly() {
Review Comment:
getDisablePreemptionForObserveOnly -> getPreemptionObserveOnly
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1693,6 +1726,14 @@ public boolean getIntraQueuePreemptionDisabled(String queue,
+ QUEUE_PREEMPTION_DISABLED, defaultVal);
}
+ public void setDisablePreemptionForPreemtionModes(String preemptionType, boolean value) {
+ setBoolean(PREEMPTION_CONFIG_PREFIX + preemptionType, value);
+ }
Review Comment:
The property observe only is a valid one, however it's not connected to disabling preemption itself, so the naming is misleading. And observe_only is a property of preemption, I wouldn't call it a type. Also please don't expect the property name as a parameter, because it destroys the purpose of the setter.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1794,29 +1835,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
- DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ public void setQueueMaxParallelApplications(String value) {
+ set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setQueueMaxParallelApplications(String queue, String value) {
Review Comment:
It's generally a good idea to name the getters and setters exactly the same apart from the prefix: getMaxParallelAppsForQueue -> setMaxParallelAppsForQueue
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1794,29 +1835,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
- DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ public void setQueueMaxParallelApplications(String value) {
+ set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setQueueMaxParallelApplications(String queue, String value) {
+ set(getQueuePrefix(queue) + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setUserMaxParallelApplications(String value) {
+ set(PREFIX + "user." + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setUserMaxParallelApplications(String user, int value) {
Review Comment:
setUserMaxParallelApplications -> setMaxParallelAppsForUser
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1794,29 +1835,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
- DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ public void setQueueMaxParallelApplications(String value) {
+ set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setQueueMaxParallelApplications(String queue, String value) {
+ set(getQueuePrefix(queue) + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setUserMaxParallelApplications(String value) {
Review Comment:
Same as my comment at setQueueMaxParallelApplications: this could be renamed to setDefaultMaxParallelAppsPerUser
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1203,6 +1218,16 @@ public void reinitializeConfigurationProperties() {
configurationProperties = new ConfigurationProperties(props);
}
+ public void setQueueMaximumAllocationMb(String queue, String value) {
+ String queuePrefix = getQueuePrefix(queue);
+ set(queuePrefix + "maximum-allocation-mb", value);
+ }
+
+ public void setQueueMaximumAllocationVcores(String queue, String value) {
+ String queuePrefix = getQueuePrefix(queue);
+ set(queuePrefix + "maximum-allocation-vcores", value);
Review Comment:
Same as above.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1794,29 +1835,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
- DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ public void setQueueMaxParallelApplications(String value) {
+ set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setQueueMaxParallelApplications(String queue, String value) {
+ set(getQueuePrefix(queue) + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setUserMaxParallelApplications(String value) {
+ set(PREFIX + "user." + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setUserMaxParallelApplications(String user, int value) {
+ setInt(getUserPrefix(user) + MAX_PARALLEL_APPLICATIONS, value);
+ }
+ public Integer getMaxParallelAppsForQueue(String queue) {
String maxParallelAppsForQueue = get(getQueuePrefix(queue)
+ MAX_PARALLEL_APPLICATIONS);
return (maxParallelAppsForQueue != null) ?
Integer.parseInt(maxParallelAppsForQueue)
- : defaultMaxParallelAppsForQueue;
+ : getMaxParallelAppsForQueue();
}
- public Integer getMaxParallelAppsForUser(String user) {
- int defaultMaxParallelAppsForUser =
- getInt(PREFIX + "user." + MAX_PARALLEL_APPLICATIONS,
+ public Integer getMaxParallelAppsForQueue() {
+ return getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ }
Review Comment:
Just like the setter, the getters could be renamed accordingly.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1794,29 +1835,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
- DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ public void setQueueMaxParallelApplications(String value) {
Review Comment:
Since this property is essentially the fallback default value of the per queue max parallel applications I suggest renaming it accordingly. I.e: setDefaultMaxParallelAppsPerQueue
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1794,29 +1835,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
- DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ public void setQueueMaxParallelApplications(String value) {
+ set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setQueueMaxParallelApplications(String queue, String value) {
+ set(getQueuePrefix(queue) + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setUserMaxParallelApplications(String value) {
+ set(PREFIX + "user." + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public void setUserMaxParallelApplications(String user, int value) {
+ setInt(getUserPrefix(user) + MAX_PARALLEL_APPLICATIONS, value);
+ }
+ public Integer getMaxParallelAppsForQueue(String queue) {
String maxParallelAppsForQueue = get(getQueuePrefix(queue)
+ MAX_PARALLEL_APPLICATIONS);
return (maxParallelAppsForQueue != null) ?
Integer.parseInt(maxParallelAppsForQueue)
- : defaultMaxParallelAppsForQueue;
+ : getMaxParallelAppsForQueue();
}
- public Integer getMaxParallelAppsForUser(String user) {
- int defaultMaxParallelAppsForUser =
- getInt(PREFIX + "user." + MAX_PARALLEL_APPLICATIONS,
+ public Integer getMaxParallelAppsForQueue() {
+ return getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ }
+
+ public Integer getMaxParallelAppsForUser(String user) {
String maxParallelAppsForUser = get(getUserPrefix(user)
+ MAX_PARALLEL_APPLICATIONS);
return (maxParallelAppsForUser != null) ?
Integer.parseInt(maxParallelAppsForUser)
- : defaultMaxParallelAppsForUser;
+ : getMaxParallelAppsForUser();
+ }
+
+ public Integer getMaxParallelAppsForUser() {
+ return getInt(PREFIX + "user." + MAX_PARALLEL_APPLICATIONS,
+ DEFAULT_MAX_PARALLEL_APPLICATIONS);
}
Review Comment:
Same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1419306055
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 7s | | trunk passed |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 6s | | trunk passed |
| -1 :x: | javadoc | 0m 58s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 38s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 34 unchanged - 0 fixed = 38 total (was 34) |
| +1 :green_heart: | mvnsite | 0m 53s | | the patch passed |
| -1 :x: | javadoc | 0m 39s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 57s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 24m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 98m 57s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 207m 3s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForQueue(String) At CapacitySchedulerConfiguration.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForQueue(String) At CapacitySchedulerConfiguration.java:[line 1881] |
| | Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForUser(String) At CapacitySchedulerConfiguration.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForUser(String) At CapacitySchedulerConfiguration.java:[line 1868] |
| Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSQueueConverter |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b1db648d9965 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c9f171226bdbf281a6cd8462a78036427342db6f |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/testReport/ |
| Max. process+thread count | 946 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1660169676
Thanks @susheel-gupta, the latest state LGTM. Can you please fix the checkstyle issue and that line ending?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "susheel-gupta (via GitHub)" <gi...@apache.org>.
susheel-gupta commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1097183443
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -723,6 +727,17 @@ public <S extends SchedulableEntity> OrderingPolicy<S> getAppOrderingPolicy(
return orderingPolicy;
}
+ public void setOrderingPolicy(String queue,
+ String appOrderingPolicy, String postfix, boolean value) {
+ setBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, value);
+ }
+
+ public boolean getOrderingPolicy(String queue, String appOrderingPolicy, String postfix) {
+ return getBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, false);
+ }
Review Comment:
Yes, the naming is misleading(for now I have removed this getter and setter), if we don't introduce a getter and setters for this then the purpose of this the jira seems to be incomplete. So what do you suggest, do we go without any getter and setter for the above property.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087784312
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -508,29 +521,29 @@ public void testReservationSystemNotSupported() {
}
private void assertNoValueForQueues(Set<String> queues, String postfix,
- Configuration config) {
+ CapacitySchedulerConfiguration capacitySchedulerConfig) {
for (String queue : queues) {
String key = PREFIX + queue + postfix;
assertNull("Key " + key + " has value, but it should be null",
- config.get(key));
+ capacitySchedulerConfig.get(key));
Review Comment:
I think we should also check where we are calling this method, because in some cases the simple get should be changed to the corresponding convenience method here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1645708912
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 31m 19s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 31m 37s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 28s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 33 unchanged - 0 fixed = 34 total (was 33) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 34s | | the patch passed |
| -1 :x: | shadedclient | 29m 21s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 19s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/7/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| +0 :ok: | asflicense | 0m 28s | | ASF License check generated no output? |
| | | 111m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f28e358b7a46 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b048b10ed3534407fffc6970180c42f440cdbd47 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/7/testReport/ |
| Max. process+thread count | 756 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1660551459
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 35s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 53s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 86m 0s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 175m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c53b65f809ad 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f98c387125a6e3bbe4ffb4010a68df379d0dbf32 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/9/testReport/ |
| Max. process+thread count | 938 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "susheel-gupta (via GitHub)" <gi...@apache.org>.
susheel-gupta commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1405059886
> I added some comments where I saw further possibilities for using the convenience get methods beside the fixed ones. I also saw that there are some config settings in the FsQueueConverter class where the CapacitySchedulerConfiguration’s setter methods could be used too, so that way we would have only one place for setting these values.
>
> For example here we can use the CapacitySchedulerConfiguration’s setMaximumCapacity() method:
>
> https://github.com/apache/hadoop/blob/a38eb1f19318e3dd34d9312afe1096db46652f49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java#L167
>
> What are your thoughts on that @brumi1024, @susheel-gupta?
I think the setters can be handled in another separate jira. What do you suggest @p-szucs @brumi1024 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087770675
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##########
@@ -214,31 +216,31 @@ public void testDefaultMaxAMShareDisabled() throws Exception {
converter.convert(params);
- Configuration conf = converter.getCapacitySchedulerConfig();
+ CapacitySchedulerConfiguration conf = converter.getCapacitySchedulerConfig();
// -1.0 means disabled ==> 1.0 in CS
assertEquals("Default max-am-resource-percent", "1.0",
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT));
- // root.admins.bob -1.0 equals to the default -1.0
- assertNull("root.admins.bob maximum-am-resource-percent should be null",
- conf.get(PREFIX + "root.admins.bob.maximum-am-resource-percent"));
+ // root.admins.bob is unset,so falling back to the global value
+ assertEquals("root.admins.bob maximum-am-resource-percent", 1.0f,
+ conf.getMaximumApplicationMasterResourcePerQueuePercent("root.admins.bob"), 0.0f);
// root.admins.alice 0.15 != -1.0
- assertEquals("root.admins.alice max-am-resource-percent", "0.15",
- conf.get(PREFIX + "root.admins.alice.maximum-am-resource-percent"));
+ assertEquals("root.admins.alice max-am-resource-percent", 0.15f,
+ conf.getMaximumApplicationMasterResourcePerQueuePercent("root.admins.alice"), 0.0f);
- // root.users.joe is unset, inherits -1.0
- assertNull("root.users.joe maximum-am-resource-percent should be null",
- conf.get(PREFIX + "root.users.joe.maximum-am-resource-percent"));
+ // root.users.joe is unset,so falling back to the global value
+ assertEquals("root.users.joe maximum-am-resource-percent", 1.0f,
+ conf.getMaximumApplicationMasterResourcePerQueuePercent("root.users.joe"), 0.0f);
}
@Test
public void testConvertACLs() throws Exception {
converter.convert(config);
- Configuration conf = converter.getCapacitySchedulerConfig();
+ CapacitySchedulerConfiguration conf = converter.getCapacitySchedulerConfig();
// root
assertEquals("root submit ACL", "alice,bob,joe,john hadoop_users",
Review Comment:
There is a getAcl() method in the config class which you can use in this test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1097291596
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -723,6 +727,17 @@ public <S extends SchedulableEntity> OrderingPolicy<S> getAppOrderingPolicy(
return orderingPolicy;
}
+ public void setOrderingPolicy(String queue,
+ String appOrderingPolicy, String postfix, boolean value) {
+ setBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, value);
+ }
+
+ public boolean getOrderingPolicy(String queue, String appOrderingPolicy, String postfix) {
+ return getBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, false);
+ }
Review Comment:
Ordering policy is pluggable, meaning that if someone wants to write a custom one they can, and they can use any type of properties under it. Hence I think adding a boolean getter is misleading, we could simply avoid writing getters for these, as they're not too widely used.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -723,6 +727,17 @@ public <S extends SchedulableEntity> OrderingPolicy<S> getAppOrderingPolicy(
return orderingPolicy;
}
+ public void setOrderingPolicy(String queue,
+ String appOrderingPolicy, String postfix, boolean value) {
+ setBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, value);
+ }
+
+ public boolean getOrderingPolicy(String queue, String appOrderingPolicy, String postfix) {
+ return getBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, false);
+ }
Review Comment:
Ordering policy is pluggable, meaning that if someone wants to write a custom one they can, and they can use any type of properties under it. Hence I think adding a boolean getter is misleading, we could simply avoid writing getters/setters for these, as they're not too widely used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1083831502
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -150,19 +152,19 @@ public void testConvertQueueHierarchy() {
// root children
assertEquals("root children", "admins,users,misc,default",
- csConfig.get(PREFIX + "root.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.queues"));
Review Comment:
There is a helper method called getQueues, which returns the list of queues that could be used here.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -150,19 +152,19 @@ public void testConvertQueueHierarchy() {
// root children
assertEquals("root children", "admins,users,misc,default",
- csConfig.get(PREFIX + "root.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.queues"));
// root.admins children
assertEquals("root.admins children", "bob,alice",
- csConfig.get(PREFIX + "root.admins.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.admins.queues"));
Review Comment:
Same as above.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -182,17 +184,18 @@ public void testQueueMaxAMShare() {
// root.admins.bob
assertEquals("root.admins.bob AM share", "1.0",
- csConfig.get(PREFIX + "root.admins.bob.maximum-am-resource-percent"));
+ capacitySchedulerConfig.get(
Review Comment:
getMaximumAMResourcePercentPerPartition checks this correctly if queried for No Label label.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -182,17 +184,18 @@ public void testQueueMaxAMShare() {
// root.admins.bob
assertEquals("root.admins.bob AM share", "1.0",
- csConfig.get(PREFIX + "root.admins.bob.maximum-am-resource-percent"));
+ capacitySchedulerConfig.get(
+ PREFIX + "root.admins.bob.maximum-am-resource-percent"));
// root.admins.alice
assertEquals("root.admins.alice AM share", "0.15",
- csConfig.get(PREFIX +
- "root.admins.alice.maximum-am-resource-percent"));
+ capacitySchedulerConfig.get(
Review Comment:
Same as the previous comment.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -218,21 +222,26 @@ public void testQueueMaxAllocations() {
// root.admins vcores + mb
assertEquals("root.admins max vcores", 3,
- csConfig.getInt(PREFIX + "root.admins.maximum-allocation-vcores", -1));
+ capacitySchedulerConfig.getInt(
+ PREFIX + "root.admins.maximum-allocation-vcores", -1));
assertEquals("root.admins max memory", 4096,
- csConfig.getInt(PREFIX + "root.admins.maximum-allocation-mb", -1));
+ capacitySchedulerConfig.getInt(
Review Comment:
getQueueMaximumAllocationMb is a helper method for this.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -242,15 +251,16 @@ public void testQueuePreemptionDisabled() {
converter.convertQueueHierarchy(rootQueue);
assertTrue("root.admins.alice preemption setting",
- csConfig.getBoolean(PREFIX + "root.admins.alice.disable_preemption",
- false));
+ capacitySchedulerConfig.getBoolean(
+ PREFIX + "root.admins.alice.disable_preemption", false));
assertTrue("root.users.joe preemption setting",
- csConfig.getBoolean(PREFIX + "root.users.joe.disable_preemption",
- false));
+ capacitySchedulerConfig.getBoolean(
Review Comment:
Same as above.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -202,12 +205,13 @@ public void testQueueMaxParallelApps() {
converter.convertQueueHierarchy(rootQueue);
assertEquals("root.admins.alice max apps", 2,
- csConfig.getInt(PREFIX + "root.admins.alice.max-parallel-apps",
- -1));
+ capacitySchedulerConfig.getInt(
Review Comment:
Please use getMaxParallelAppsForQueue.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -218,21 +222,26 @@ public void testQueueMaxAllocations() {
// root.admins vcores + mb
assertEquals("root.admins max vcores", 3,
- csConfig.getInt(PREFIX + "root.admins.maximum-allocation-vcores", -1));
+ capacitySchedulerConfig.getInt(
Review Comment:
getQueueMaximumAllocationVcores returns this.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -150,19 +152,19 @@ public void testConvertQueueHierarchy() {
// root children
assertEquals("root children", "admins,users,misc,default",
- csConfig.get(PREFIX + "root.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.queues"));
// root.admins children
assertEquals("root.admins children", "bob,alice",
- csConfig.get(PREFIX + "root.admins.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.admins.queues"));
// root.default children - none
- assertNull("root.default children", csConfig.get(PREFIX + "root.default" +
- ".queues"));
+ assertNull("root.default children",
+ capacitySchedulerConfig.get(PREFIX + "root.default" + ".queues"));
Review Comment:
Same as above.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -150,19 +152,19 @@ public void testConvertQueueHierarchy() {
// root children
assertEquals("root children", "admins,users,misc,default",
- csConfig.get(PREFIX + "root.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.queues"));
// root.admins children
assertEquals("root.admins children", "bob,alice",
- csConfig.get(PREFIX + "root.admins.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.admins.queues"));
// root.default children - none
- assertNull("root.default children", csConfig.get(PREFIX + "root.default" +
- ".queues"));
+ assertNull("root.default children",
+ capacitySchedulerConfig.get(PREFIX + "root.default" + ".queues"));
// root.users children
assertEquals("root.users children", "john,joe",
- csConfig.get(PREFIX + "root.users.queues"));
+ capacitySchedulerConfig.get(PREFIX + "root.users.queues"));
Review Comment:
Same as above.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -218,21 +222,26 @@ public void testQueueMaxAllocations() {
// root.admins vcores + mb
assertEquals("root.admins max vcores", 3,
- csConfig.getInt(PREFIX + "root.admins.maximum-allocation-vcores", -1));
+ capacitySchedulerConfig.getInt(
+ PREFIX + "root.admins.maximum-allocation-vcores", -1));
assertEquals("root.admins max memory", 4096,
- csConfig.getInt(PREFIX + "root.admins.maximum-allocation-mb", -1));
+ capacitySchedulerConfig.getInt(
+ PREFIX + "root.admins.maximum-allocation-mb", -1));
// root.users.john max vcores + mb
assertEquals("root.users.john max vcores", 2,
- csConfig.getInt(PREFIX + "root.users.john.maximum-allocation-vcores",
- -1));
+ capacitySchedulerConfig.getInt(
Review Comment:
Same as the comment before the last one.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -242,15 +251,16 @@ public void testQueuePreemptionDisabled() {
converter.convertQueueHierarchy(rootQueue);
assertTrue("root.admins.alice preemption setting",
- csConfig.getBoolean(PREFIX + "root.admins.alice.disable_preemption",
- false));
+ capacitySchedulerConfig.getBoolean(
Review Comment:
getPreemptionDisabled helper method checks this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1400097285
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 33s | | trunk passed |
| +1 :green_heart: | compile | 1m 5s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| -1 :x: | javadoc | 0m 56s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/1/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 52s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 37s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 53s | | the patch passed |
| -1 :x: | javadoc | 0m 38s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/1/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 2s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 98m 54s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 206m 35s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 18008820a202 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b196d224f2db4ca0b1762be055d65c606df3906d |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/1/testReport/ |
| Max. process+thread count | 953 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1633290629
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 57s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 35m 20s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 45s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 33 unchanged - 0 fixed = 37 total (was 33) |
| +1 :green_heart: | mvnsite | 0m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 26s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 100m 13s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 235m 20s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7ccbcc1a8ccf 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4de2e19125ebedd03c649d869b3b92273da897d6 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/6/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "susheel-gupta (via GitHub)" <gi...@apache.org>.
susheel-gupta commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1086487425
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -149,20 +152,20 @@ public void testConvertQueueHierarchy() {
converter.convertQueueHierarchy(rootQueue);
// root children
- assertEquals("root children", "admins,users,misc,default",
- csConfig.get(PREFIX + "root.queues"));
+ assertEquals("root children", new String[]{"admins", "users", "misc", "default"},
Review Comment:
I will update this with assertArrayEquals as I was unaware of assertEquals is deprecated for array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087778061
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##########
@@ -440,28 +454,27 @@ public void testQueueOrderingPolicy() throws Exception {
rootQueue = fs.getQueueManager().getRootQueue();
converter.convertQueueHierarchy(rootQueue);
-
// root
assertEquals("root ordering policy", null,
- csConfig.get(PREFIX + "root.ordering-policy"));
+ capacitySchedulerConfig.get(PREFIX + "root.ordering-policy"));
Review Comment:
Could you please use the getAppOrderingPolicy() method to get the configs in this test case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1405165777
> > I added some comments where I saw further possibilities for using the convenience get methods beside the fixed ones. I also saw that there are some config settings in the FsQueueConverter class where the CapacitySchedulerConfiguration’s setter methods could be used too, so that way we would have only one place for setting these values.
> > For example here we can use the CapacitySchedulerConfiguration’s setMaximumCapacity() method:
> > https://github.com/apache/hadoop/blob/a38eb1f19318e3dd34d9312afe1096db46652f49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java#L167
> >
> > What are your thoughts on that @brumi1024, @susheel-gupta?
>
> I think the setters can be handled in another separate jira. What do you suggest @p-szucs @brumi1024 ?
I agree with @p-szucs, it strongly connects to this PR, so it be best to do it in one go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1410567489
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 57s | | trunk passed |
| +1 :green_heart: | compile | 1m 7s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 4s | | trunk passed |
| -1 :x: | javadoc | 0m 58s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 25s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 39s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 34 unchanged - 0 fixed = 38 total (was 34) |
| +1 :green_heart: | mvnsite | 0m 53s | | the patch passed |
| -1 :x: | javadoc | 0m 38s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 58s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 24m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 98m 43s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 207m 35s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| | Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForQueue(String) At CapacitySchedulerConfiguration.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForQueue(String) At CapacitySchedulerConfiguration.java:[line 1873] |
| | Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForUser(String) At CapacitySchedulerConfiguration.java:then immediately reboxed in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getMaxParallelAppsForUser(String) At CapacitySchedulerConfiguration.java:[line 1887] |
| Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSQueueConverter |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux daea1b01a867 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3b56d6879bdfeebdfb5000a9d4712dcb153174e6 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/testReport/ |
| Max. process+thread count | 950 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1664499358
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 8m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 56s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 51s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 8s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 29s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 85m 54s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 183m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c966823d9900 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e07fd03c9b2a1f97026e8db121c6f17274e28e22 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/10/testReport/ |
| Max. process+thread count | 958 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1665223248
Thanks @susheel-gupta for the contribution, @p-szucs for the review, merging to trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 merged pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 merged PR #5320:
URL: https://github.com/apache/hadoop/pull/5320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087764208
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##########
@@ -214,31 +216,31 @@ public void testDefaultMaxAMShareDisabled() throws Exception {
converter.convert(params);
- Configuration conf = converter.getCapacitySchedulerConfig();
+ CapacitySchedulerConfiguration conf = converter.getCapacitySchedulerConfig();
// -1.0 means disabled ==> 1.0 in CS
assertEquals("Default max-am-resource-percent", "1.0",
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT));
Review Comment:
The same as my previous comment.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##########
@@ -214,31 +216,31 @@ public void testDefaultMaxAMShareDisabled() throws Exception {
converter.convert(params);
- Configuration conf = converter.getCapacitySchedulerConfig();
+ CapacitySchedulerConfiguration conf = converter.getCapacitySchedulerConfig();
// -1.0 means disabled ==> 1.0 in CS
assertEquals("Default max-am-resource-percent", "1.0",
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT));
Review Comment:
The same here as my previous comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1270365720
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1816,29 +1846,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
+ public void setDefaultMaxParallelAppsPerQueue(String value) {
Review Comment:
The naming is misleading, I would simply use setDefaultMaxParallelApps as there is a queue based MaxParallelApps setting.
Also same as with the maximumAllocation properties: this should get an int parameter and use setInt.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -501,8 +501,12 @@ public int getMaximumSystemApplications() {
return maxApplications;
}
+ public void setMaximumApplicationMasterResourcePercent(float percent) {
+ setFloat(PREFIX + MAXIMUM_AM_RESOURCE_SUFFIX, percent);
+ }
+
public float getMaximumApplicationMasterResourcePercent() {
- return getFloat(MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT,
+ return getFloat(PREFIX + MAXIMUM_AM_RESOURCE_SUFFIX,
Review Comment:
Is this change needed? If MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT exists (and has quite a few uses in tests) why not use it?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1816,29 +1846,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
+ public void setDefaultMaxParallelAppsPerQueue(String value) {
+ set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public Integer getDefaultMaxParallelAppsPerQueue() {
Review Comment:
Same as above, getDefaultMaxParallelApps.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1222,6 +1226,16 @@ public void reinitializeConfigurationProperties() {
configurationProperties = new ConfigurationProperties(props);
}
+ public void setQueueMaximumAllocationMb(String queue, String value) {
+ String queuePrefix = getQueuePrefix(queue);
+ set(queuePrefix + MAXIMUM_ALLOCATION_MB, value);
Review Comment:
If the getter returns an int the setter should accept an int. I would use int value parameter and setInt method. Same for below.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -1816,29 +1846,48 @@ public static boolean shouldAppFailFast(Configuration conf) {
return conf.getBoolean(APP_FAIL_FAST, DEFAULT_APP_FAIL_FAST);
}
- public Integer getMaxParallelAppsForQueue(String queue) {
- int defaultMaxParallelAppsForQueue =
- getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
+ public void setDefaultMaxParallelAppsPerQueue(String value) {
+ set(PREFIX + MAX_PARALLEL_APPLICATIONS, value);
+ }
+
+ public Integer getDefaultMaxParallelAppsPerQueue() {
+ return getInt(PREFIX + MAX_PARALLEL_APPLICATIONS,
DEFAULT_MAX_PARALLEL_APPLICATIONS);
+ }
- String maxParallelAppsForQueue = get(getQueuePrefix(queue)
- + MAX_PARALLEL_APPLICATIONS);
+ public void setDefaultMaxParallelAppsPerUser(String value) {
Review Comment:
setInt, and int parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1645797483
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 21s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 39s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/8/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 27s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 33 unchanged - 0 fixed = 34 total (was 33) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 85m 20s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 175m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f7cec7a2438c 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bb96d26b6ae6201c54f23dfe32b8bef0af32f493 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/8/testReport/ |
| Max. process+thread count | 935 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087762346
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##########
@@ -172,37 +170,41 @@ private void createConverter() {
public void testDefaultMaxAMShare() throws Exception {
converter.convert(config);
- Configuration conf = converter.getCapacitySchedulerConfig();
+ CapacitySchedulerConfiguration conf = converter.getCapacitySchedulerConfig();
String maxAmShare =
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT);
Review Comment:
There is a getMaximumApplicationMasterResourcePercent() method in the CapacitySchedulerConfiuration class which returns this value. Could you please use that method here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1403399970
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 0s | | trunk passed |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| -1 :x: | javadoc | 0m 54s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 0m 56s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 3 new + 227 unchanged - 0 fixed = 230 total (was 227) |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 0m 50s | [/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 3 new + 184 unchanged - 0 fixed = 187 total (was 184) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 38s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| -1 :x: | javadoc | 0m 38s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 98m 33s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 209m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 81451809fd6b 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a38eb1f19318e3dd34d9312afe1096db46652f49 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/testReport/ |
| Max. process+thread count | 939 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087762346
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##########
@@ -172,37 +170,41 @@ private void createConverter() {
public void testDefaultMaxAMShare() throws Exception {
converter.convert(config);
- Configuration conf = converter.getCapacitySchedulerConfig();
+ CapacitySchedulerConfiguration conf = converter.getCapacitySchedulerConfig();
String maxAmShare =
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT);
Review Comment:
There is a getMaximumApplicationMasterResourcePercent() method in the CapacitySchedulerConfiuration class which returns this value. Could you please use that here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] p-szucs commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "p-szucs (via GitHub)" <gi...@apache.org>.
p-szucs commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1404951874
I added some comments where I saw further possibilities for using the convenience get methods beside the fixed ones. I also saw that there are some config settings in the FsQueueConverter class where the CapacitySchedulerConfiguration’s setter methods could be used too, so that way we would have only one place for setting these values.
For example here we can use the CapacitySchedulerConfiguration’s setMaximumCapacity() method: https://github.com/apache/hadoop/blob/a38eb1f19318e3dd34d9312afe1096db46652f49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java#L167
What are your thoughts on that @brumi1024, @susheel-gupta?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on a diff in pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
Posted by "susheel-gupta (via GitHub)" <gi...@apache.org>.
susheel-gupta commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1097183443
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java:
##########
@@ -723,6 +727,17 @@ public <S extends SchedulableEntity> OrderingPolicy<S> getAppOrderingPolicy(
return orderingPolicy;
}
+ public void setOrderingPolicy(String queue,
+ String appOrderingPolicy, String postfix, boolean value) {
+ setBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, value);
+ }
+
+ public boolean getOrderingPolicy(String queue, String appOrderingPolicy, String postfix) {
+ return getBoolean(getQueuePrefix(queue) +
+ "ordering-policy" + DOT + appOrderingPolicy + DOT + postfix, false);
+ }
Review Comment:
Yes, the naming is misleading(for now I have removed this getter and setter), if we don't introduce a getter and setters for this then the purpose of this the jira seems to be incomplete. So what do you suggest, do we go without this getter and setter for the above property.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org