You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/07/28 15:43:07 UTC
[GitHub] [hadoop] slfan1989 opened a new pull request, #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
slfan1989 opened a new pull request, #4656:
URL: https://github.com/apache/hadoop/pull/4656
JIRA:YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r936069438
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/GetSubClustersInfoResponse.java:
##########
@@ -63,4 +74,13 @@ public static GetSubClustersInfoResponse newInstance(
@Unstable
public abstract void setSubClusters(List<SubClusterInfo> subClusters);
+ /**
+ * Set the Collection of {@link SubClusterInfo} representing the information about
+ * all sub-clusters that are currently participating in Federation.
+ *
+ * @param subClusters the list of {@link SubClusterInfo}
+ */
+ @Private
+ @Unstable
+ public abstract void setSubClusters(Collection<SubClusterInfo> subClusters);
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r935003118
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/BaseRouterPoliciesTest.java:
##########
@@ -115,4 +117,14 @@ public void testAllBlacklistSubcluster() throws YarnException {
}
}
}
+
+ @Test
+ public void testNullReservationContext() throws Exception {
+ FederationRouterPolicy policy = ((FederationRouterPolicy) getPolicy());
Review Comment:
I will fix it.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestLoadBasedRouterPolicy.java:
##########
@@ -46,12 +46,15 @@ public void setUp() throws Exception {
Map<SubClusterIdInfo, Float> routerWeights = new HashMap<>();
Map<SubClusterIdInfo, Float> amrmWeights = new HashMap<>();
+ long now = System.currentTimeMillis();
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1200160822
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 12s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 108m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f02f8a41b06e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b258c9a98923fb68c157a7ad7f9173a30d4d604e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/4/testReport/ |
| Max. process+thread count | 592 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r935124487
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationPoliciesTestUtil.java:
##########
@@ -159,6 +159,50 @@ public static void initializePolicyContext(
new Configuration());
}
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters,
+ FederationStateStoreFacade facade) throws YarnException {
+ FederationPolicyInitializationContext context =
+ new FederationPolicyInitializationContext(null, initResolver(), facade,
+ SubClusterId.newInstance("homesubcluster"));
+ return initializePolicyContext2(context, policy, policyInfo, activeSubClusters);
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ return initializePolicyContext2(policy, policyInfo, activeSubClusters, initFacade());
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ FederationPolicyInitializationContext fpc,
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ ByteBuffer buf = policyInfo.toByteBuffer();
+ fpc.setSubClusterPolicyConfiguration(SubClusterPolicyConfiguration
+ .newInstance("queue1", policy.getClass().getCanonicalName(), buf));
+
+ if (fpc.getFederationStateStoreFacade() == null) {
+ FederationStateStoreFacade facade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = mock(FederationStateStore.class);
+
+ if (activeSubClusters == null) {
+ activeSubClusters = new HashMap<>();
+ }
+ GetSubClustersInfoResponse response =
Review Comment:
Thank you for your help reviewing the code. I will modify the code
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationPoliciesTestUtil.java:
##########
@@ -159,6 +159,50 @@ public static void initializePolicyContext(
new Configuration());
}
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters,
+ FederationStateStoreFacade facade) throws YarnException {
+ FederationPolicyInitializationContext context =
+ new FederationPolicyInitializationContext(null, initResolver(), facade,
+ SubClusterId.newInstance("homesubcluster"));
+ return initializePolicyContext2(context, policy, policyInfo, activeSubClusters);
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ return initializePolicyContext2(policy, policyInfo, activeSubClusters, initFacade());
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ FederationPolicyInitializationContext fpc,
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ ByteBuffer buf = policyInfo.toByteBuffer();
+ fpc.setSubClusterPolicyConfiguration(SubClusterPolicyConfiguration
+ .newInstance("queue1", policy.getClass().getCanonicalName(), buf));
+
+ if (fpc.getFederationStateStoreFacade() == null) {
+ FederationStateStoreFacade facade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = mock(FederationStateStore.class);
+
+ if (activeSubClusters == null) {
+ activeSubClusters = new HashMap<>();
+ }
+ GetSubClustersInfoResponse response =
Review Comment:
Thank you for your help reviewing the code. I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r937196261
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java:
##########
@@ -175,8 +175,7 @@ private SubClusterId getRandomActiveSubCluster(
}
List<SubClusterId> list = new ArrayList<>(activeSubclusters.keySet());
- FederationPolicyUtils.validateSubClusterAvailability(
- list, blackListSubClusters);
+ FederationPolicyUtils.validateSubClusterAvailability(list, blackListSubClusters);
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r936075841
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/GetSubClustersInfoResponse.java:
##########
@@ -43,6 +44,16 @@ public static GetSubClustersInfoResponse newInstance(
return subClusterInfos;
}
+ @Public
+ @Unstable
+ public static GetSubClustersInfoResponse newInstance(
+ Collection<SubClusterInfo> subClusters) {
+ GetSubClustersInfoResponse subClusterInfos =
+ Records.newRecord(GetSubClustersInfoResponse.class);
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1203371708
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 22s | | trunk passed |
| +1 :green_heart: | compile | 4m 10s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 28s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 13s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 48s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 3m 58s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 58s | | the patch passed |
| +1 :green_heart: | compile | 3m 19s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 15s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 26s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 2m 45s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/8/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 2m 32s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/8/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt) | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 133m 20s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.federation.policies.manager.TestHomePolicyManager |
| | hadoop.yarn.server.federation.policies.amrmproxy.TestLocalityMulticastAMRMProxyPolicy |
| | hadoop.yarn.server.federation.policies.router.TestUniformRandomRouterPolicy |
| | hadoop.yarn.server.federation.policies.TestFederationPolicyInitializationContextValidator |
| | hadoop.yarn.server.federation.policies.router.TestLocalityRouterPolicy |
| | hadoop.yarn.server.federation.store.impl.TestMemoryFederationStateStore |
| | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| | hadoop.yarn.server.federation.policies.amrmproxy.TestHomeAMRMProxyPolicy |
| | hadoop.yarn.server.federation.policies.router.TestHashBasedRouterPolicy |
| | hadoop.yarn.server.federation.policies.router.TestPriorityRouterPolicy |
| | hadoop.yarn.server.federation.policies.manager.TestWeightedLocalityPolicyManager |
| | hadoop.yarn.server.federation.policies.manager.TestPriorityBroadcastPolicyManager |
| | hadoop.yarn.server.federation.store.impl.TestZookeeperFederationStateStore |
| | hadoop.yarn.server.federation.policies.manager.TestRejectAllPolicyManager |
| | hadoop.yarn.server.federation.utils.TestFederationStateStoreFacade |
| | hadoop.yarn.server.federation.policies.amrmproxy.TestBroadcastAMRMProxyFederationPolicy |
| | hadoop.yarn.server.federation.policies.router.TestRejectRouterPolicy |
| | hadoop.yarn.server.federation.policies.manager.TestUniformBroadcastPolicyManager |
| | hadoop.yarn.server.federation.policies.router.TestWeightedRandomRouterPolicy |
| | hadoop.yarn.server.federation.policies.TestRouterPolicyFacade |
| | hadoop.yarn.server.federation.policies.manager.TestHashBasedBroadcastPolicyManager |
| | hadoop.yarn.server.federation.policies.amrmproxy.TestRejectAMRMProxyPolicy |
| | hadoop.yarn.server.federation.policies.router.TestLoadBasedRouterPolicy |
| | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptorRetry |
| | hadoop.yarn.server.router.webapp.TestFederationInterceptorRESTRetry |
| | hadoop.yarn.server.router.webapp.TestFederationInterceptorREST |
| | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9e0e9cafa95a 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 28b10e947e58c02eb5df45ce4e2f7d14d64d99e4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/8/testReport/ |
| Max. process+thread count | 711 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1203455347
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 17m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 21s | | trunk passed |
| +1 :green_heart: | compile | 4m 10s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 28s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 34s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 3s | | the patch passed |
| +1 :green_heart: | compile | 4m 4s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 4s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 12s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 23s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 48s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 3m 25s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 134m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1277fa604050 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62dd3a1b558a3be649a49d42c40d836a41163ad8 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/9/testReport/ |
| Max. process+thread count | 1741 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933043084
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/WeightedRandomRouterPolicy.java:
##########
@@ -35,36 +35,21 @@
* sub-clusters.
*/
public class WeightedRandomRouterPolicy extends AbstractRouterPolicy {
-
@Override
- public SubClusterId getHomeSubcluster(
- ApplicationSubmissionContext appSubmissionContext,
- List<SubClusterId> blacklist) throws YarnException {
-
- // null checks and default-queue behavior
- validate(appSubmissionContext);
-
- Map<SubClusterId, SubClusterInfo> activeSubclusters =
- getActiveSubclusters();
+ protected SubClusterId chooseSubCluster(
+ String queue, Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException {
- FederationPolicyUtils.validateSubClusterAvailability(
- new ArrayList<SubClusterId>(activeSubclusters.keySet()), blacklist);
-
- // note: we cannot pre-compute the weights, as the set of activeSubcluster
+ // note: we cannot pre-compute the weights, as the set of activeSubCluster
// changes dynamically (and this would unfairly spread the load to
// sub-clusters adjacent to an inactive one), hence we need to count/scan
// the list and based on weight pick the next sub-cluster.
Map<SubClusterIdInfo, Float> weights =
- getPolicyInfo().getRouterPolicyWeights();
+ getPolicyInfo().getRouterPolicyWeights();
ArrayList<Float> weightList = new ArrayList<>();
ArrayList<SubClusterId> scIdList = new ArrayList<>();
for (Map.Entry<SubClusterIdInfo, Float> entry : weights.entrySet()) {
- if (blacklist != null && blacklist.contains(entry.getKey().toId())) {
- continue;
- }
- if (entry.getKey() != null
- && activeSubclusters.containsKey(entry.getKey().toId())) {
+ if (entry.getKey() != null && preSelectSubClusters.containsKey(entry.getKey().toId())) {
Review Comment:
Thank you for your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933042759
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/WeightedRandomRouterPolicy.java:
##########
@@ -35,36 +35,21 @@
* sub-clusters.
*/
public class WeightedRandomRouterPolicy extends AbstractRouterPolicy {
-
@Override
- public SubClusterId getHomeSubcluster(
- ApplicationSubmissionContext appSubmissionContext,
- List<SubClusterId> blacklist) throws YarnException {
-
- // null checks and default-queue behavior
- validate(appSubmissionContext);
-
- Map<SubClusterId, SubClusterInfo> activeSubclusters =
- getActiveSubclusters();
+ protected SubClusterId chooseSubCluster(
+ String queue, Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException {
- FederationPolicyUtils.validateSubClusterAvailability(
- new ArrayList<SubClusterId>(activeSubclusters.keySet()), blacklist);
-
- // note: we cannot pre-compute the weights, as the set of activeSubcluster
+ // note: we cannot pre-compute the weights, as the set of activeSubCluster
// changes dynamically (and this would unfairly spread the load to
// sub-clusters adjacent to an inactive one), hence we need to count/scan
// the list and based on weight pick the next sub-cluster.
Map<SubClusterIdInfo, Float> weights =
Review Comment:
I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1202662451
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 25m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 30s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| -1 :x: | compile | 0m 31s | [/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 0m 30s | [/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -0 :warning: | checkstyle | 0m 28s | [/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | The patch fails to run checkstyle in hadoop-yarn-server |
| -1 :x: | mvnsite | 0m 32s | [/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in trunk failed. |
| -1 :x: | mvnsite | 0m 30s | [/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt) | hadoop-yarn-server-router in trunk failed. |
| -1 :x: | javadoc | 0m 31s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server-common in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javadoc | 0m 31s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server-router in trunk failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javadoc | 0m 32s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server-common in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javadoc | 0m 30s | [/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server-router in trunk failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | spotbugs | 0m 30s | [/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in trunk failed. |
| -1 :x: | spotbugs | 0m 31s | [/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt) | hadoop-yarn-server-router in trunk failed. |
| -1 :x: | shadedclient | 12m 37s | | branch has errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 46s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 52s | | the patch passed |
| -1 :x: | compile | 0m 30s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 0m 30s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 0m 31s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 0m 31s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/blanks-eol.txt) | The patch has 47 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/blanks-tabs.txt) | The patch 400 line(s) with tabs. |
| -0 :warning: | checkstyle | 0m 29s | [/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | The patch fails to run checkstyle in hadoop-yarn-server |
| +1 :green_heart: | mvnsite | 1m 50s | | the patch passed |
| -1 :x: | javadoc | 0m 59s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 48 new + 0 unchanged - 0 fixed = 48 total (was 0) |
| -1 :x: | javadoc | 0m 37s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) |
| -1 :x: | javadoc | 0m 41s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 48 new + 0 unchanged - 0 fixed = 48 total (was 0) |
| -1 :x: | javadoc | 0m 33s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) |
| +1 :green_heart: | spotbugs | 3m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 47s | | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 3m 26s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt) | hadoop-yarn-server-router in the patch passed. |
| -1 :x: | asflicense | 0m 45s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 90m 52s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8d0fe6528bb0 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fd81fc177cbd410447d737d28c5fbf82fec987b9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/testReport/ |
| Max. process+thread count | 1928 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/6/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1200053097
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 23s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 21 new + 1 unchanged - 0 fixed = 22 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 37s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 2m 47s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 105m 53s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.federation.policies.router.TestPriorityRouterPolicy |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e71971f7c799 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5412975396eaff893fa1ac1db2ff2efc08b31bff |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/3/testReport/ |
| Max. process+thread count | 540 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r935933008
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestPriorityRouterPolicy.java:
##########
@@ -54,10 +55,11 @@ public void setUp() throws Exception {
// with 5% omit a subcluster
if (getRand().nextFloat() < 0.95f || i == 5) {
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
- getActiveSubclusters().put(sc.toId(), sci);
+ long now = Time.now();
+ SubClusterInfo federationSubClusterInfo =
Review Comment:
Not sure the indetation is correct.
Let's do:
```
SubClusterInfo federationSubClusterInfo = SubClusterInfo.newInstance(
sc.toId(), "dns1:80", "dns1:81", "dns1:82", "dns1:83",
now - 1000, SubClusterState.SC_RUNNING, now - 2000, generateClusterMetricsInfo(i));
```
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/GetSubClustersInfoResponse.java:
##########
@@ -43,6 +44,16 @@ public static GetSubClustersInfoResponse newInstance(
return subClusterInfos;
}
+ @Public
+ @Unstable
+ public static GetSubClustersInfoResponse newInstance(
+ Collection<SubClusterInfo> subClusters) {
+ GetSubClustersInfoResponse subClusterInfos =
+ Records.newRecord(GetSubClustersInfoResponse.class);
Review Comment:
Indentation doesn't seem correct.
I think it even fits in one line.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationPoliciesTestUtil.java:
##########
@@ -159,6 +159,50 @@ public static void initializePolicyContext(
new Configuration());
}
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters,
+ FederationStateStoreFacade facade) throws YarnException {
+ FederationPolicyInitializationContext context =
+ new FederationPolicyInitializationContext(null, initResolver(), facade,
+ SubClusterId.newInstance("homesubcluster"));
+ return initializePolicyContext2(context, policy, policyInfo, activeSubClusters);
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ return initializePolicyContext2(policy, policyInfo, activeSubClusters, initFacade());
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ FederationPolicyInitializationContext fpc,
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ ByteBuffer buf = policyInfo.toByteBuffer();
+ fpc.setSubClusterPolicyConfiguration(SubClusterPolicyConfiguration
+ .newInstance("queue1", policy.getClass().getCanonicalName(), buf));
+
+ if (fpc.getFederationStateStoreFacade() == null) {
+ FederationStateStoreFacade facade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = mock(FederationStateStore.class);
+
+ if (activeSubClusters == null) {
+ activeSubClusters = new HashMap<>();
+ }
+ GetSubClustersInfoResponse response =
Review Comment:
One line?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/BaseFederationPoliciesTest.java:
##########
@@ -177,11 +185,64 @@ public void setHomeSubCluster(SubClusterId homeSubCluster) {
public void setMockActiveSubclusters(int numSubclusters) {
for (int i = 1; i <= numSubclusters; i++) {
SubClusterIdInfo sc = new SubClusterIdInfo("sc" + i);
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
+ SubClusterInfo sci = SubClusterInfo.newInstance(sc.toId(),
+ "dns1:80", "dns1:81", "dns1:82", "dns1:83", SubClusterState.SC_RUNNING,
+ System.currentTimeMillis(), "something");
getActiveSubclusters().put(sc.toId(), sci);
}
}
+ public String generateClusterMetricsInfo(int id) {
+ long mem = 1024 * getRand().nextInt(277 * 100 - 1);
+ // plant a best cluster
+ if (id == 5) {
+ mem = 1024 * 277 * 100;
+ }
+ String clusterMetrics =
Review Comment:
The indentation looks too aggressive, and you could use the 100 chars limit better.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/GetSubClustersInfoResponse.java:
##########
@@ -63,4 +74,13 @@ public static GetSubClustersInfoResponse newInstance(
@Unstable
public abstract void setSubClusters(List<SubClusterInfo> subClusters);
+ /**
+ * Set the Collection of {@link SubClusterInfo} representing the information about
+ * all sub-clusters that are currently participating in Federation.
+ *
+ * @param subClusters the list of {@link SubClusterInfo}
+ */
+ @Private
+ @Unstable
+ public abstract void setSubClusters(Collection<SubClusterInfo> subClusters);
Review Comment:
I don't think we need to add a new one, we can just replace the List one.
As the type is more generic, we should have this be backwards compatible.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestWeightedRandomRouterPolicy.java:
##########
@@ -68,10 +67,12 @@ public void configureWeights(float numSubClusters) {
SubClusterIdInfo sc = new SubClusterIdInfo("sc" + i);
// with 5% omit a subcluster
if (getRand().nextFloat() < 0.95f) {
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
- getActiveSubclusters().put(sc.toId(), sci);
+ long now = Time.now();
+ SubClusterInfo federationSubClusterInfo =
Review Comment:
Same as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933793535
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/LocalityRouterPolicy.java:
##########
@@ -128,53 +126,55 @@ public SubClusterId getHomeSubcluster(
ResourceRequest nodeRequest = null;
ResourceRequest rackRequest = null;
ResourceRequest anyRequest = null;
+
for (ResourceRequest rr : rrList) {
// Handle "node" requests
try {
targetId = resolver.getSubClusterForNode(rr.getResourceName());
nodeRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve node : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve node.");
Review Comment:
I will fix it and use e.getMessage() .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r934901461
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/BaseFederationPoliciesTest.java:
##########
@@ -177,11 +185,65 @@ public void setHomeSubCluster(SubClusterId homeSubCluster) {
public void setMockActiveSubclusters(int numSubclusters) {
for (int i = 1; i <= numSubclusters; i++) {
SubClusterIdInfo sc = new SubClusterIdInfo("sc" + i);
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
+ SubClusterInfo sci = SubClusterInfo.newInstance(sc.toId(),
+ "dns1:80", "dns1:81", "dns1:82", "dns1:83", SubClusterState.SC_RUNNING,
+ System.currentTimeMillis(), "something");
getActiveSubclusters().put(sc.toId(), sci);
}
}
+ public String generateClusterMetricsInfo(int id) {
+ long mem = 1024 * getRand().nextInt(277 * 100 - 1);
+ // plant a best cluster
+ if (id == 5) {
+ mem = 1024 * 277 * 100;
+ }
+ String clusterMetrics =
+ "{\"clusterMetrics\":{\"appsSubmitted\":65," + "\"appsCompleted\":64,"
+ + "\"appsPending\":0,\"appsRunning\":0,\"appsFailed\":0,"
+ + "\"appsKilled\":1,\"reservedMB\":0,\"availableMB\":" + mem + ","
+ + "\"allocatedMB\":0,\"reservedVirtualCores\":0,"
+ + "\"availableVirtualCores\":2216,\"allocatedVirtualCores\":0,"
+ + "\"containersAllocated\":0,\"containersReserved\":0,"
+ + "\"containersPending\":0,\"totalMB\":28364800,"
+ + "\"totalVirtualCores\":2216,\"totalNodes\":278,\"lostNodes\":1,"
+ + "\"unhealthyNodes\":0,\"decommissionedNodes\":0,"
+ + "\"rebootedNodes\":0,\"activeNodes\":277}}\n";
+
+ return clusterMetrics;
+ }
+
+ public FederationStateStoreFacade getMemoryFacade() throws YarnException {
+
+ // setting up a store and its facade (with caching off)
+ FederationStateStoreFacade fedFacade =
Review Comment:
One line
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/BaseFederationPoliciesTest.java:
##########
@@ -177,11 +185,65 @@ public void setHomeSubCluster(SubClusterId homeSubCluster) {
public void setMockActiveSubclusters(int numSubclusters) {
for (int i = 1; i <= numSubclusters; i++) {
SubClusterIdInfo sc = new SubClusterIdInfo("sc" + i);
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
+ SubClusterInfo sci = SubClusterInfo.newInstance(sc.toId(),
+ "dns1:80", "dns1:81", "dns1:82", "dns1:83", SubClusterState.SC_RUNNING,
+ System.currentTimeMillis(), "something");
getActiveSubclusters().put(sc.toId(), sci);
}
}
+ public String generateClusterMetricsInfo(int id) {
+ long mem = 1024 * getRand().nextInt(277 * 100 - 1);
+ // plant a best cluster
+ if (id == 5) {
+ mem = 1024 * 277 * 100;
+ }
+ String clusterMetrics =
+ "{\"clusterMetrics\":{\"appsSubmitted\":65," + "\"appsCompleted\":64,"
+ + "\"appsPending\":0,\"appsRunning\":0,\"appsFailed\":0,"
+ + "\"appsKilled\":1,\"reservedMB\":0,\"availableMB\":" + mem + ","
+ + "\"allocatedMB\":0,\"reservedVirtualCores\":0,"
+ + "\"availableVirtualCores\":2216,\"allocatedVirtualCores\":0,"
+ + "\"containersAllocated\":0,\"containersReserved\":0,"
+ + "\"containersPending\":0,\"totalMB\":28364800,"
+ + "\"totalVirtualCores\":2216,\"totalNodes\":278,\"lostNodes\":1,"
+ + "\"unhealthyNodes\":0,\"decommissionedNodes\":0,"
+ + "\"rebootedNodes\":0,\"activeNodes\":277}}\n";
+
+ return clusterMetrics;
+ }
+
+ public FederationStateStoreFacade getMemoryFacade() throws YarnException {
+
+ // setting up a store and its facade (with caching off)
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ YarnConfiguration conf = new YarnConfiguration();
+ conf.set(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS, "0");
Review Comment:
setInt?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/AbstractRouterPolicy.java:
##########
@@ -63,4 +71,108 @@ public void validate(ApplicationSubmissionContext appSubmissionContext)
}
}
+ /**
+ * This method is implemented by the specific policy, and it is used to route
+ * both reservations, and applications among a given set of
+ * sub-clusters.
+ *
+ * @param queue the queue for this application/reservation
+ * @param preSelectSubClusters a pre-filter set of sub-clusters
+ * @return the chosen sub-cluster
+ *
+ * @throws YarnException if the policy fails to choose a sub-cluster
+ */
+ protected abstract SubClusterId chooseSubCluster(String queue,
+ Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException;
+
+ /**
+ * Filter chosen SubCluster based on reservationId.
+ *
+ * @param reservationId the globally unique identifier for a reservation.
+ * @param activeSubClusters the map of ids to info for all active subclusters.
+ * @return the chosen sub-cluster
+ * @throws YarnException if the policy fails to choose a sub-cluster
+ */
+ protected Map<SubClusterId, SubClusterInfo> prefilterSubClusters(
+ ReservationId reservationId, Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+
+ // if a reservation exists limit scope to the sub-cluster this
+ // reservation is mapped to
+ // TODO: Implemented in YARN-11236
+ return activeSubClusters;
+ }
+
+ /**
+ * Simply picks from alphabetically-sorted active subclusters based on the
+ * hash of quey name. Jobs of the same queue will all be routed to the same
+ * sub-cluster, as far as the number of active sub-cluster and their names
+ * remain the same.
+ *
+ * @param appContext the {@link ApplicationSubmissionContext} that
+ * has to be routed to an appropriate subCluster for execution.
+ *
+ * @param blackLists the list of subClusters as identified by
+ * {@link SubClusterId} to blackList from the selection of the home
+ * subCluster.
+ *
+ * @return a hash-based chosen {@link SubClusterId} that will be the "home"
+ * for this application.
+ *
+ * @throws YarnException if there are no active subclusters.
+ */
+ @Override
+ public SubClusterId getHomeSubcluster(ApplicationSubmissionContext appContext,
+ List<SubClusterId> blackLists) throws YarnException {
+
+ // null checks and default-queue behavior
+ validate(appContext);
+
+ // apply filtering based on reservation location and active sub-clusters
+ Map<SubClusterId, SubClusterInfo> filteredSubClusters = prefilterSubClusters(
+ appContext.getReservationID(), getActiveSubclusters());
+
+ FederationPolicyUtils.validateSubClusterAvailability(
Review Comment:
Can we have `validateSubClusterAvailability` taking Collection so we don't need temp data structures?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestLoadBasedRouterPolicy.java:
##########
@@ -46,12 +46,15 @@ public void setUp() throws Exception {
Map<SubClusterIdInfo, Float> routerWeights = new HashMap<>();
Map<SubClusterIdInfo, Float> amrmWeights = new HashMap<>();
+ long now = System.currentTimeMillis();
Review Comment:
Time.now()?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestPriorityRouterPolicy.java:
##########
@@ -54,10 +54,11 @@ public void setUp() throws Exception {
// with 5% omit a subcluster
if (getRand().nextFloat() < 0.95f || i == 5) {
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
- getActiveSubclusters().put(sc.toId(), sci);
+ long now = System.currentTimeMillis();
Review Comment:
Time.now()
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/BaseRouterPoliciesTest.java:
##########
@@ -115,4 +117,14 @@ public void testAllBlacklistSubcluster() throws YarnException {
}
}
}
+
+ @Test
+ public void testNullReservationContext() throws Exception {
+ FederationRouterPolicy policy = ((FederationRouterPolicy) getPolicy());
Review Comment:
Too many spaces after the =
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestLoadBasedRouterPolicy.java:
##########
@@ -131,7 +133,7 @@ public void testIfNoSubclustersWithWeightOne() {
fail();
} catch (YarnException ex) {
Assert.assertTrue(
Review Comment:
LambdaTestUtils?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationPoliciesTestUtil.java:
##########
@@ -159,6 +159,50 @@ public static void initializePolicyContext(
new Configuration());
}
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters,
+ FederationStateStoreFacade facade) throws YarnException {
+ FederationPolicyInitializationContext context =
+ new FederationPolicyInitializationContext(null, initResolver(), facade,
+ SubClusterId.newInstance("homesubcluster"));
+ return initializePolicyContext2(context, policy, policyInfo, activeSubClusters);
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ return initializePolicyContext2(policy, policyInfo, activeSubClusters, initFacade());
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ FederationPolicyInitializationContext fpc,
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ ByteBuffer buf = policyInfo.toByteBuffer();
+ fpc.setSubClusterPolicyConfiguration(SubClusterPolicyConfiguration
+ .newInstance("queue1", policy.getClass().getCanonicalName(), buf));
+
+ if (fpc.getFederationStateStoreFacade() == null) {
+ FederationStateStoreFacade facade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = mock(FederationStateStore.class);
+
+ if (activeSubClusters == null) {
+ activeSubClusters = new HashMap<>();
+ }
+ GetSubClustersInfoResponse response =
Review Comment:
Can we create a generic version of GetSubClustersInfoResponse.newInstance?
One that takes Collection<String>
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/BaseRouterPoliciesTest.java:
##########
@@ -36,6 +37,7 @@
import org.junit.Assert;
import org.junit.Test;
+
Review Comment:
Avoid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r935042136
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationPoliciesTestUtil.java:
##########
@@ -159,6 +159,50 @@ public static void initializePolicyContext(
new Configuration());
}
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters,
+ FederationStateStoreFacade facade) throws YarnException {
+ FederationPolicyInitializationContext context =
+ new FederationPolicyInitializationContext(null, initResolver(), facade,
+ SubClusterId.newInstance("homesubcluster"));
+ return initializePolicyContext2(context, policy, policyInfo, activeSubClusters);
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ return initializePolicyContext2(policy, policyInfo, activeSubClusters, initFacade());
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ FederationPolicyInitializationContext fpc,
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ ByteBuffer buf = policyInfo.toByteBuffer();
+ fpc.setSubClusterPolicyConfiguration(SubClusterPolicyConfiguration
+ .newInstance("queue1", policy.getClass().getCanonicalName(), buf));
+
+ if (fpc.getFederationStateStoreFacade() == null) {
+ FederationStateStoreFacade facade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = mock(FederationStateStore.class);
+
+ if (activeSubClusters == null) {
+ activeSubClusters = new HashMap<>();
+ }
+ GetSubClustersInfoResponse response =
Review Comment:
This?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933442715
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/PriorityRouterPolicy.java:
##########
@@ -17,61 +17,40 @@
package org.apache.hadoop.yarn.server.federation.policies.router;
-import java.util.ArrayList;
-import java.util.List;
import java.util.Map;
-import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.exceptions.YarnException;
-import org.apache.hadoop.yarn.server.federation.policies.FederationPolicyUtils;
import org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyException;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
/**
* This implements a policy that interprets "weights" as a ordered list of
- * preferences among sub-clusters. Highest weight among active subclusters is
+ * preferences among sub-clusters. Highest weight among active subClusters is
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r932565594
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/AbstractRouterPolicy.java:
##########
@@ -63,4 +71,109 @@ public void validate(ApplicationSubmissionContext appSubmissionContext)
}
}
+ /**
+ * This method is implemented by the specific policy, and it is used to route
+ * both reservations, and applications among a given set of
+ * sub-clusters.
+ *
+ * @param queue the queue for this application/reservation
+ * @param preSelectSubClusters a pre-filter set of sub-clusters
+ * @return the chosen sub-cluster
+ *
+ * @throws YarnException if the policy fails to choose a sub-cluster
+ */
+ protected abstract SubClusterId chooseSubCluster(String queue,
+ Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException;
+
+ /**
+ * Filter chosen SubCluster based on reservationId.
+ *
+ * @param reservationId the globally unique identifier for a reservation.
+ * @param activeSubClusters the map of ids to info for all active subclusters.
+ * @return the chosen sub-cluster
+ * @throws YarnException if the policy fails to choose a sub-cluster
+ */
+ protected Map<SubClusterId, SubClusterInfo> prefilterSubClusters(
+ ReservationId reservationId, Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+
+ // if a reservation exists limit scope to the sub-cluster this
+ // reservation is mapped to
+ // TODO: Implemented in YARN-11236
+ return activeSubClusters;
+ }
+
+ /**
+ * Simply picks from alphabetically-sorted active subclusters based on the
+ * hash of quey name. Jobs of the same queue will all be routed to the same
+ * sub-cluster, as far as the number of active sub-cluster and their names
+ * remain the same.
+ *
+ * @param appContext the {@link ApplicationSubmissionContext} that
+ * has to be routed to an appropriate subCluster for execution.
+ *
+ * @param blackLists the list of subClusters as identified by
+ * {@link SubClusterId} to blackList from the selection of the home
+ * subCluster.
+ *
+ * @return a hash-based chosen {@link SubClusterId} that will be the "home"
+ * for this application.
+ *
+ * @throws YarnException if there are no active subclusters.
+ */
+ @Override
+ public SubClusterId getHomeSubcluster(ApplicationSubmissionContext appContext,
+ List<SubClusterId> blackLists) throws YarnException {
+
+ // null checks and default-queue behavior
+ validate(appContext);
+
+ // apply filtering based on reservation location and active sub-clusters
+ Map<SubClusterId, SubClusterInfo> filteredSubClusters = prefilterSubClusters(
+ appContext.getReservationID(), getActiveSubclusters());
+
+ FederationPolicyUtils.validateSubClusterAvailability(
+ new ArrayList<>(filteredSubClusters.keySet()), blackLists);
+
+ // remove black SubCluster
+ if (blackLists != null) {
+ blackLists.forEach(filteredSubClusters::remove);
+ }
+
+ // pick the chosen subCluster from the active ones
+ return chooseSubCluster(appContext.getQueue(), filteredSubClusters);
+ }
+
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ @Override
+ public SubClusterId getReservationHomeSubcluster(ReservationSubmissionRequest request)
+ throws YarnException {
+ if (request == null) {
+ throw new FederationPolicyException(
Review Comment:
Single line?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/WeightedRandomRouterPolicy.java:
##########
@@ -35,36 +35,21 @@
* sub-clusters.
*/
public class WeightedRandomRouterPolicy extends AbstractRouterPolicy {
-
@Override
- public SubClusterId getHomeSubcluster(
- ApplicationSubmissionContext appSubmissionContext,
- List<SubClusterId> blacklist) throws YarnException {
-
- // null checks and default-queue behavior
- validate(appSubmissionContext);
-
- Map<SubClusterId, SubClusterInfo> activeSubclusters =
- getActiveSubclusters();
+ protected SubClusterId chooseSubCluster(
+ String queue, Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException {
- FederationPolicyUtils.validateSubClusterAvailability(
- new ArrayList<SubClusterId>(activeSubclusters.keySet()), blacklist);
-
- // note: we cannot pre-compute the weights, as the set of activeSubcluster
+ // note: we cannot pre-compute the weights, as the set of activeSubCluster
// changes dynamically (and this would unfairly spread the load to
// sub-clusters adjacent to an inactive one), hence we need to count/scan
// the list and based on weight pick the next sub-cluster.
Map<SubClusterIdInfo, Float> weights =
Review Comment:
single line
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/UniformRandomRouterPolicy.java:
##########
@@ -59,46 +60,13 @@ public void reinitialize(FederationPolicyInitializationContext policyContext)
setPolicyContext(policyContext);
}
- /**
- * Simply picks a random active subCluster to start the AM (this does NOT
- * depend on the weights in the policy).
- *
- * @param appSubmissionContext the {@link ApplicationSubmissionContext} that
- * has to be routed to an appropriate subCluster for execution.
- *
- * @param blackListSubClusters the list of subClusters as identified by
- * {@link SubClusterId} to blackList from the selection of the home
- * subCluster.
- *
- * @return a randomly chosen subcluster.
- *
- * @throws YarnException if there are no active subclusters.
- */
@Override
- public SubClusterId getHomeSubcluster(
- ApplicationSubmissionContext appSubmissionContext,
- List<SubClusterId> blackListSubClusters) throws YarnException {
-
- // null checks and default-queue behavior
- validate(appSubmissionContext);
-
- Map<SubClusterId, SubClusterInfo> activeSubclusters =
- getActiveSubclusters();
-
- List<SubClusterId> list = new ArrayList<>(activeSubclusters.keySet());
-
- FederationPolicyUtils.validateSubClusterAvailability(list,
- blackListSubClusters);
-
- if (blackListSubClusters != null) {
-
- // Remove from the active SubClusters from StateStore the blacklisted ones
- for (SubClusterId scId : blackListSubClusters) {
- list.remove(scId);
- }
+ protected SubClusterId chooseSubCluster(
+ String queue, Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException {
+ if(preSelectSubClusters == null || preSelectSubClusters.size() == 0) {
Review Comment:
space after if
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/LocalityRouterPolicy.java:
##########
@@ -128,53 +126,55 @@ public SubClusterId getHomeSubcluster(
ResourceRequest nodeRequest = null;
ResourceRequest rackRequest = null;
ResourceRequest anyRequest = null;
+
for (ResourceRequest rr : rrList) {
// Handle "node" requests
try {
targetId = resolver.getSubClusterForNode(rr.getResourceName());
nodeRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve node : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve node.", e);
Review Comment:
I prefer not having the full exception. Thoughts?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/BaseFederationPoliciesTest.java:
##########
@@ -28,20 +28,21 @@
import java.util.Map;
import java.util.Random;
+import org.apache.hadoop.yarn.api.protocolrecords.ReservationSubmissionRequest;
import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.server.federation.policies.amrmproxy.FederationAMRMProxyPolicy;
import org.apache.hadoop.yarn.server.federation.policies.dao.WeightedPolicyInfo;
import org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyException;
import org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyInitializationException;
import org.apache.hadoop.yarn.server.federation.policies.router.FederationRouterPolicy;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.*;
Review Comment:
Avoid
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/WeightedRandomRouterPolicy.java:
##########
@@ -35,36 +35,21 @@
* sub-clusters.
*/
public class WeightedRandomRouterPolicy extends AbstractRouterPolicy {
-
@Override
- public SubClusterId getHomeSubcluster(
- ApplicationSubmissionContext appSubmissionContext,
- List<SubClusterId> blacklist) throws YarnException {
-
- // null checks and default-queue behavior
- validate(appSubmissionContext);
-
- Map<SubClusterId, SubClusterInfo> activeSubclusters =
- getActiveSubclusters();
+ protected SubClusterId chooseSubCluster(
+ String queue, Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException {
- FederationPolicyUtils.validateSubClusterAvailability(
- new ArrayList<SubClusterId>(activeSubclusters.keySet()), blacklist);
-
- // note: we cannot pre-compute the weights, as the set of activeSubcluster
+ // note: we cannot pre-compute the weights, as the set of activeSubCluster
// changes dynamically (and this would unfairly spread the load to
// sub-clusters adjacent to an inactive one), hence we need to count/scan
// the list and based on weight pick the next sub-cluster.
Map<SubClusterIdInfo, Float> weights =
- getPolicyInfo().getRouterPolicyWeights();
+ getPolicyInfo().getRouterPolicyWeights();
ArrayList<Float> weightList = new ArrayList<>();
ArrayList<SubClusterId> scIdList = new ArrayList<>();
for (Map.Entry<SubClusterIdInfo, Float> entry : weights.entrySet()) {
- if (blacklist != null && blacklist.contains(entry.getKey().toId())) {
- continue;
- }
- if (entry.getKey() != null
- && activeSubclusters.containsKey(entry.getKey().toId())) {
+ if (entry.getKey() != null && preSelectSubClusters.containsKey(entry.getKey().toId())) {
Review Comment:
Extract.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/WeightedRandomRouterPolicy.java:
##########
@@ -73,9 +58,8 @@ public SubClusterId getHomeSubcluster(
int pickedIndex = FederationPolicyUtils.getWeightedRandom(weightList);
if (pickedIndex == -1) {
throw new FederationPolicyException(
- "No positive weight found on active subclusters");
+ "No positive weight found on active subClusters.");
Review Comment:
i would leave the old capitalization
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/UniformRandomRouterPolicy.java:
##########
@@ -59,46 +60,13 @@ public void reinitialize(FederationPolicyInitializationContext policyContext)
setPolicyContext(policyContext);
}
- /**
- * Simply picks a random active subCluster to start the AM (this does NOT
- * depend on the weights in the policy).
- *
- * @param appSubmissionContext the {@link ApplicationSubmissionContext} that
- * has to be routed to an appropriate subCluster for execution.
- *
- * @param blackListSubClusters the list of subClusters as identified by
- * {@link SubClusterId} to blackList from the selection of the home
- * subCluster.
- *
- * @return a randomly chosen subcluster.
- *
- * @throws YarnException if there are no active subclusters.
- */
@Override
- public SubClusterId getHomeSubcluster(
- ApplicationSubmissionContext appSubmissionContext,
- List<SubClusterId> blackListSubClusters) throws YarnException {
-
- // null checks and default-queue behavior
- validate(appSubmissionContext);
-
- Map<SubClusterId, SubClusterInfo> activeSubclusters =
- getActiveSubclusters();
-
- List<SubClusterId> list = new ArrayList<>(activeSubclusters.keySet());
-
- FederationPolicyUtils.validateSubClusterAvailability(list,
- blackListSubClusters);
-
- if (blackListSubClusters != null) {
-
- // Remove from the active SubClusters from StateStore the blacklisted ones
- for (SubClusterId scId : blackListSubClusters) {
- list.remove(scId);
- }
+ protected SubClusterId chooseSubCluster(
+ String queue, Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException {
+ if(preSelectSubClusters == null || preSelectSubClusters.size() == 0) {
Review Comment:
isEmpty()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933412600
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/RouterPolicyFacade.java:
##########
@@ -262,4 +219,92 @@ public synchronized void reset() {
}
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ public SubClusterId getReservationHomeSubCluster(
+ ReservationSubmissionRequest request) throws YarnException {
+
+ // the maps are concurrent, but we need to protect from reset()
+ // reinitialization mid-execution by creating a new reference local to this
+ // method.
+ Map<String, SubClusterPolicyConfiguration> cachedConfs = globalConfMap;
+ Map<String, FederationRouterPolicy> policyMap = globalPolicyMap;
+
+ if (request == null) {
+ throw new FederationPolicyException(
+ "The ReservationSubmissionRequest cannot be null.");
+ }
+
+ String queue = request.getQueue();
+ FederationRouterPolicy policy = getFederationRouterPolicy(cachedConfs, policyMap, queue);
+
+ if (policy == null) {
+ // this should never happen, as the to maps are updated together
+ throw new FederationPolicyException("No FederationRouterPolicy found "
+ + "for queue: " + request.getQueue() + " (while routing "
+ + "reservation: " + request.getReservationId() + ") "
+ + "and no default specified.");
+ }
+
+ return policy.getReservationHomeSubcluster(request);
+ }
+
+ private FederationRouterPolicy getFederationRouterPolicy(
+ Map<String, SubClusterPolicyConfiguration> cachedConfiguration,
+ Map<String, FederationRouterPolicy> policyMap, String queue)
+ throws FederationPolicyInitializationException {
+
+ // the facade might cache this request, based on its parameterization
+ SubClusterPolicyConfiguration configuration = null;
+ String copyQueue = queue;
+
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("There is no policy configured for the queue: {}, " +
Review Comment:
Thanks for your help reviewing the code, I will modify the code.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/RouterPolicyFacade.java:
##########
@@ -262,4 +219,92 @@ public synchronized void reset() {
}
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ public SubClusterId getReservationHomeSubCluster(
+ ReservationSubmissionRequest request) throws YarnException {
+
+ // the maps are concurrent, but we need to protect from reset()
+ // reinitialization mid-execution by creating a new reference local to this
+ // method.
+ Map<String, SubClusterPolicyConfiguration> cachedConfs = globalConfMap;
+ Map<String, FederationRouterPolicy> policyMap = globalPolicyMap;
+
+ if (request == null) {
+ throw new FederationPolicyException(
+ "The ReservationSubmissionRequest cannot be null.");
+ }
+
+ String queue = request.getQueue();
+ FederationRouterPolicy policy = getFederationRouterPolicy(cachedConfs, policyMap, queue);
+
+ if (policy == null) {
+ // this should never happen, as the to maps are updated together
+ throw new FederationPolicyException("No FederationRouterPolicy found "
+ + "for queue: " + request.getQueue() + " (while routing "
+ + "reservation: " + request.getReservationId() + ") "
+ + "and no default specified.");
+ }
+
+ return policy.getReservationHomeSubcluster(request);
+ }
+
+ private FederationRouterPolicy getFederationRouterPolicy(
+ Map<String, SubClusterPolicyConfiguration> cachedConfiguration,
+ Map<String, FederationRouterPolicy> policyMap, String queue)
+ throws FederationPolicyInitializationException {
+
+ // the facade might cache this request, based on its parameterization
+ SubClusterPolicyConfiguration configuration = null;
+ String copyQueue = queue;
+
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("There is no policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+
+ // If there is no policy configured for this queue, fallback to the baseline
+ // policy that is configured either in the store or via XML config (and
+ // cached)
+ if (configuration == null) {
+ final String policyKey = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ LOG.warn("There is no policies configured for queue: {} " +
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933031522
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/LocalityRouterPolicy.java:
##########
@@ -128,53 +126,55 @@ public SubClusterId getHomeSubcluster(
ResourceRequest nodeRequest = null;
ResourceRequest rackRequest = null;
ResourceRequest anyRequest = null;
+
for (ResourceRequest rr : rrList) {
// Handle "node" requests
try {
targetId = resolver.getSubClusterForNode(rr.getResourceName());
nodeRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve node : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve node.", e);
Review Comment:
Thanks for your suggestion, I will modify the code!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1200422116
@goiri Please help to review the code again, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r937120315
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java:
##########
@@ -175,8 +175,7 @@ private SubClusterId getRandomActiveSubCluster(
}
List<SubClusterId> list = new ArrayList<>(activeSubclusters.keySet());
- FederationPolicyUtils.validateSubClusterAvailability(
- list, blackListSubClusters);
+ FederationPolicyUtils.validateSubClusterAvailability(list, blackListSubClusters);
Review Comment:
We could just do:
```
FederationPolicyUtils.validateSubClusterAvailability(activeSubclusters.keySet(), blackListSubClusters);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1205505375
@goiri Thank you for helping to review the code, I will follow up with YARN-11236 after this pr is completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933034630
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/AbstractRouterPolicy.java:
##########
@@ -63,4 +71,109 @@ public void validate(ApplicationSubmissionContext appSubmissionContext)
}
}
+ /**
+ * This method is implemented by the specific policy, and it is used to route
+ * both reservations, and applications among a given set of
+ * sub-clusters.
+ *
+ * @param queue the queue for this application/reservation
+ * @param preSelectSubClusters a pre-filter set of sub-clusters
+ * @return the chosen sub-cluster
+ *
+ * @throws YarnException if the policy fails to choose a sub-cluster
+ */
+ protected abstract SubClusterId chooseSubCluster(String queue,
+ Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException;
+
+ /**
+ * Filter chosen SubCluster based on reservationId.
+ *
+ * @param reservationId the globally unique identifier for a reservation.
+ * @param activeSubClusters the map of ids to info for all active subclusters.
+ * @return the chosen sub-cluster
+ * @throws YarnException if the policy fails to choose a sub-cluster
+ */
+ protected Map<SubClusterId, SubClusterInfo> prefilterSubClusters(
+ ReservationId reservationId, Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+
+ // if a reservation exists limit scope to the sub-cluster this
+ // reservation is mapped to
+ // TODO: Implemented in YARN-11236
+ return activeSubClusters;
+ }
+
+ /**
+ * Simply picks from alphabetically-sorted active subclusters based on the
+ * hash of quey name. Jobs of the same queue will all be routed to the same
+ * sub-cluster, as far as the number of active sub-cluster and their names
+ * remain the same.
+ *
+ * @param appContext the {@link ApplicationSubmissionContext} that
+ * has to be routed to an appropriate subCluster for execution.
+ *
+ * @param blackLists the list of subClusters as identified by
+ * {@link SubClusterId} to blackList from the selection of the home
+ * subCluster.
+ *
+ * @return a hash-based chosen {@link SubClusterId} that will be the "home"
+ * for this application.
+ *
+ * @throws YarnException if there are no active subclusters.
+ */
+ @Override
+ public SubClusterId getHomeSubcluster(ApplicationSubmissionContext appContext,
+ List<SubClusterId> blackLists) throws YarnException {
+
+ // null checks and default-queue behavior
+ validate(appContext);
+
+ // apply filtering based on reservation location and active sub-clusters
+ Map<SubClusterId, SubClusterInfo> filteredSubClusters = prefilterSubClusters(
+ appContext.getReservationID(), getActiveSubclusters());
+
+ FederationPolicyUtils.validateSubClusterAvailability(
+ new ArrayList<>(filteredSubClusters.keySet()), blackLists);
+
+ // remove black SubCluster
+ if (blackLists != null) {
+ blackLists.forEach(filteredSubClusters::remove);
+ }
+
+ // pick the chosen subCluster from the active ones
+ return chooseSubCluster(appContext.getQueue(), filteredSubClusters);
+ }
+
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ @Override
+ public SubClusterId getReservationHomeSubcluster(ReservationSubmissionRequest request)
+ throws YarnException {
+ if (request == null) {
+ throw new FederationPolicyException(
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1199426504
@goiri Please help to review the code again, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933440496
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/RouterPolicyFacade.java:
##########
@@ -262,4 +219,92 @@ public synchronized void reset() {
}
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ public SubClusterId getReservationHomeSubCluster(
+ ReservationSubmissionRequest request) throws YarnException {
+
+ // the maps are concurrent, but we need to protect from reset()
+ // reinitialization mid-execution by creating a new reference local to this
+ // method.
+ Map<String, SubClusterPolicyConfiguration> cachedConfs = globalConfMap;
+ Map<String, FederationRouterPolicy> policyMap = globalPolicyMap;
+
+ if (request == null) {
+ throw new FederationPolicyException(
+ "The ReservationSubmissionRequest cannot be null.");
+ }
+
+ String queue = request.getQueue();
+ FederationRouterPolicy policy = getFederationRouterPolicy(cachedConfs, policyMap, queue);
+
+ if (policy == null) {
+ // this should never happen, as the to maps are updated together
+ throw new FederationPolicyException("No FederationRouterPolicy found "
+ + "for queue: " + request.getQueue() + " (while routing "
+ + "reservation: " + request.getReservationId() + ") "
+ + "and no default specified.");
+ }
+
+ return policy.getReservationHomeSubcluster(request);
+ }
+
+ private FederationRouterPolicy getFederationRouterPolicy(
+ Map<String, SubClusterPolicyConfiguration> cachedConfiguration,
+ Map<String, FederationRouterPolicy> policyMap, String queue)
+ throws FederationPolicyInitializationException {
+
+ // the facade might cache this request, based on its parameterization
+ SubClusterPolicyConfiguration configuration = null;
+ String copyQueue = queue;
+
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("There is no policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+
+ // If there is no policy configured for this queue, fallback to the baseline
+ // policy that is configured either in the store or via XML config (and
+ // cached)
+ if (configuration == null) {
+ final String policyKey = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ LOG.warn("There is no policies configured for queue: {} " +
+ "we fallback to default policy for: {}. ", copyQueue, policyKey);
+ copyQueue = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("Cannot retrieve policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+ }
+
+ // the fallback is not configure via store, but via XML, using
+ // previously loaded configuration.
+ if (configuration == null) {
+ configuration = cachedConfiguration.get(YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY);
+ }
+
+ // if the configuration has changed since last loaded, reinit the policy
+ // based on current configuration
+ if (!cachedConfiguration.containsKey(copyQueue)
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933387046
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/RouterPolicyFacade.java:
##########
@@ -262,4 +219,92 @@ public synchronized void reset() {
}
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ public SubClusterId getReservationHomeSubCluster(
+ ReservationSubmissionRequest request) throws YarnException {
+
+ // the maps are concurrent, but we need to protect from reset()
+ // reinitialization mid-execution by creating a new reference local to this
+ // method.
+ Map<String, SubClusterPolicyConfiguration> cachedConfs = globalConfMap;
+ Map<String, FederationRouterPolicy> policyMap = globalPolicyMap;
+
+ if (request == null) {
+ throw new FederationPolicyException(
+ "The ReservationSubmissionRequest cannot be null.");
+ }
+
+ String queue = request.getQueue();
+ FederationRouterPolicy policy = getFederationRouterPolicy(cachedConfs, policyMap, queue);
+
+ if (policy == null) {
+ // this should never happen, as the to maps are updated together
+ throw new FederationPolicyException("No FederationRouterPolicy found "
+ + "for queue: " + request.getQueue() + " (while routing "
+ + "reservation: " + request.getReservationId() + ") "
+ + "and no default specified.");
+ }
+
+ return policy.getReservationHomeSubcluster(request);
+ }
+
+ private FederationRouterPolicy getFederationRouterPolicy(
+ Map<String, SubClusterPolicyConfiguration> cachedConfiguration,
+ Map<String, FederationRouterPolicy> policyMap, String queue)
+ throws FederationPolicyInitializationException {
+
+ // the facade might cache this request, based on its parameterization
+ SubClusterPolicyConfiguration configuration = null;
+ String copyQueue = queue;
+
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("There is no policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+
+ // If there is no policy configured for this queue, fallback to the baseline
+ // policy that is configured either in the store or via XML config (and
+ // cached)
+ if (configuration == null) {
+ final String policyKey = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ LOG.warn("There is no policies configured for queue: {} " +
Review Comment:
We can rearrange so the string is a single line.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/PriorityRouterPolicy.java:
##########
@@ -17,61 +17,40 @@
package org.apache.hadoop.yarn.server.federation.policies.router;
-import java.util.ArrayList;
-import java.util.List;
import java.util.Map;
-import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.exceptions.YarnException;
-import org.apache.hadoop.yarn.server.federation.policies.FederationPolicyUtils;
import org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyException;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
/**
* This implements a policy that interprets "weights" as a ordered list of
- * preferences among sub-clusters. Highest weight among active subclusters is
+ * preferences among sub-clusters. Highest weight among active subClusters is
Review Comment:
We keep changing the capitalization.
I think "Subcluster" is fine; no need to capitalize the C.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/RouterPolicyFacade.java:
##########
@@ -262,4 +219,92 @@ public synchronized void reset() {
}
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ public SubClusterId getReservationHomeSubCluster(
+ ReservationSubmissionRequest request) throws YarnException {
+
+ // the maps are concurrent, but we need to protect from reset()
+ // reinitialization mid-execution by creating a new reference local to this
+ // method.
+ Map<String, SubClusterPolicyConfiguration> cachedConfs = globalConfMap;
+ Map<String, FederationRouterPolicy> policyMap = globalPolicyMap;
+
+ if (request == null) {
+ throw new FederationPolicyException(
+ "The ReservationSubmissionRequest cannot be null.");
+ }
+
+ String queue = request.getQueue();
+ FederationRouterPolicy policy = getFederationRouterPolicy(cachedConfs, policyMap, queue);
+
+ if (policy == null) {
+ // this should never happen, as the to maps are updated together
+ throw new FederationPolicyException("No FederationRouterPolicy found "
+ + "for queue: " + request.getQueue() + " (while routing "
+ + "reservation: " + request.getReservationId() + ") "
+ + "and no default specified.");
+ }
+
+ return policy.getReservationHomeSubcluster(request);
+ }
+
+ private FederationRouterPolicy getFederationRouterPolicy(
+ Map<String, SubClusterPolicyConfiguration> cachedConfiguration,
+ Map<String, FederationRouterPolicy> policyMap, String queue)
+ throws FederationPolicyInitializationException {
+
+ // the facade might cache this request, based on its parameterization
+ SubClusterPolicyConfiguration configuration = null;
+ String copyQueue = queue;
+
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("There is no policy configured for the queue: {}, " +
Review Comment:
We can fit the string into a single line.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/LocalityRouterPolicy.java:
##########
@@ -128,53 +126,55 @@ public SubClusterId getHomeSubcluster(
ResourceRequest nodeRequest = null;
ResourceRequest rackRequest = null;
ResourceRequest anyRequest = null;
+
for (ResourceRequest rr : rrList) {
// Handle "node" requests
try {
targetId = resolver.getSubClusterForNode(rr.getResourceName());
nodeRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve node : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve node.");
}
// Handle "rack" requests
try {
resolver.getSubClustersForRack(rr.getResourceName());
rackRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve rack : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve rack.");
}
// Handle "ANY" requests
if (ResourceRequest.isAnyLocation(rr.getResourceName())) {
anyRequest = rr;
continue;
}
}
+
if (nodeRequest == null) {
- throw new YarnException("Missing node request");
+ throw new YarnException("Missing node request.");
}
if (rackRequest == null) {
- throw new YarnException("Missing rack request");
+ throw new YarnException("Missing rack request.");
}
if (anyRequest == null) {
- throw new YarnException("Missing any request");
+ throw new YarnException("Missing any request.");
}
- LOG.info(
- "Node request: " + nodeRequest.getResourceName() + ", Rack request: "
- + rackRequest.getResourceName() + ", Any request: " + anyRequest
- .getResourceName());
+
+ LOG.info("Node request: {} , Rack request: {} , Any request: {}.",
+ nodeRequest.getResourceName(), rackRequest.getResourceName(),
+ anyRequest.getResourceName());
+
// Handle "node" requests
if (validSubClusters.contains(targetId) && enabledSCs
.contains(targetId)) {
- LOG.info("Node {} is in SubCluster: {}", nodeRequest.getResourceName(),
- targetId);
+ LOG.info("Node {} is in SubCluster: {}.", nodeRequest.getResourceName(), targetId);
return targetId;
} else {
throw new YarnException("The node " + nodeRequest.getResourceName()
+ " is in a blacklist SubCluster or not active. ");
}
} catch (YarnException e) {
LOG.error("Validating resource requests failed, Falling back to "
- + "WeightedRandomRouterPolicy placement: " + e.getMessage());
+ + "WeightedRandomRouterPolicy placement.", e);
Review Comment:
Single line sring?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/LocalityRouterPolicy.java:
##########
@@ -128,53 +126,55 @@ public SubClusterId getHomeSubcluster(
ResourceRequest nodeRequest = null;
ResourceRequest rackRequest = null;
ResourceRequest anyRequest = null;
+
for (ResourceRequest rr : rrList) {
// Handle "node" requests
try {
targetId = resolver.getSubClusterForNode(rr.getResourceName());
nodeRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve node : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve node.");
Review Comment:
Doesn't it make sense to keep output the message at least?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/RouterPolicyFacade.java:
##########
@@ -262,4 +219,92 @@ public synchronized void reset() {
}
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ public SubClusterId getReservationHomeSubCluster(
+ ReservationSubmissionRequest request) throws YarnException {
+
+ // the maps are concurrent, but we need to protect from reset()
+ // reinitialization mid-execution by creating a new reference local to this
+ // method.
+ Map<String, SubClusterPolicyConfiguration> cachedConfs = globalConfMap;
+ Map<String, FederationRouterPolicy> policyMap = globalPolicyMap;
+
+ if (request == null) {
+ throw new FederationPolicyException(
+ "The ReservationSubmissionRequest cannot be null.");
+ }
+
+ String queue = request.getQueue();
+ FederationRouterPolicy policy = getFederationRouterPolicy(cachedConfs, policyMap, queue);
+
+ if (policy == null) {
+ // this should never happen, as the to maps are updated together
+ throw new FederationPolicyException("No FederationRouterPolicy found "
+ + "for queue: " + request.getQueue() + " (while routing "
+ + "reservation: " + request.getReservationId() + ") "
+ + "and no default specified.");
+ }
+
+ return policy.getReservationHomeSubcluster(request);
+ }
+
+ private FederationRouterPolicy getFederationRouterPolicy(
+ Map<String, SubClusterPolicyConfiguration> cachedConfiguration,
+ Map<String, FederationRouterPolicy> policyMap, String queue)
+ throws FederationPolicyInitializationException {
+
+ // the facade might cache this request, based on its parameterization
+ SubClusterPolicyConfiguration configuration = null;
+ String copyQueue = queue;
+
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("There is no policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+
+ // If there is no policy configured for this queue, fallback to the baseline
+ // policy that is configured either in the store or via XML config (and
+ // cached)
+ if (configuration == null) {
+ final String policyKey = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ LOG.warn("There is no policies configured for queue: {} " +
+ "we fallback to default policy for: {}. ", copyQueue, policyKey);
+ copyQueue = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("Cannot retrieve policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+ }
+
+ // the fallback is not configure via store, but via XML, using
+ // previously loaded configuration.
+ if (configuration == null) {
+ configuration = cachedConfiguration.get(YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY);
+ }
+
+ // if the configuration has changed since last loaded, reinit the policy
+ // based on current configuration
+ if (!cachedConfiguration.containsKey(copyQueue)
Review Comment:
Fix indentation
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/RouterPolicyFacade.java:
##########
@@ -262,4 +219,92 @@ public synchronized void reset() {
}
+ /**
+ * This method provides a wrapper of all policy functionalities for routing a
+ * reservation. Internally it manages configuration changes, and policy
+ * init/reinit.
+ *
+ * @param request the reservation to route.
+ *
+ * @return the id of the subcluster that will be the "home" for this
+ * reservation.
+ *
+ * @throws YarnException if there are issues initializing policies, or no
+ * valid sub-cluster id could be found for this reservation.
+ */
+ public SubClusterId getReservationHomeSubCluster(
+ ReservationSubmissionRequest request) throws YarnException {
+
+ // the maps are concurrent, but we need to protect from reset()
+ // reinitialization mid-execution by creating a new reference local to this
+ // method.
+ Map<String, SubClusterPolicyConfiguration> cachedConfs = globalConfMap;
+ Map<String, FederationRouterPolicy> policyMap = globalPolicyMap;
+
+ if (request == null) {
+ throw new FederationPolicyException(
+ "The ReservationSubmissionRequest cannot be null.");
+ }
+
+ String queue = request.getQueue();
+ FederationRouterPolicy policy = getFederationRouterPolicy(cachedConfs, policyMap, queue);
+
+ if (policy == null) {
+ // this should never happen, as the to maps are updated together
+ throw new FederationPolicyException("No FederationRouterPolicy found "
+ + "for queue: " + request.getQueue() + " (while routing "
+ + "reservation: " + request.getReservationId() + ") "
+ + "and no default specified.");
+ }
+
+ return policy.getReservationHomeSubcluster(request);
+ }
+
+ private FederationRouterPolicy getFederationRouterPolicy(
+ Map<String, SubClusterPolicyConfiguration> cachedConfiguration,
+ Map<String, FederationRouterPolicy> policyMap, String queue)
+ throws FederationPolicyInitializationException {
+
+ // the facade might cache this request, based on its parameterization
+ SubClusterPolicyConfiguration configuration = null;
+ String copyQueue = queue;
+
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("There is no policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+
+ // If there is no policy configured for this queue, fallback to the baseline
+ // policy that is configured either in the store or via XML config (and
+ // cached)
+ if (configuration == null) {
+ final String policyKey = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ LOG.warn("There is no policies configured for queue: {} " +
+ "we fallback to default policy for: {}. ", copyQueue, policyKey);
+ copyQueue = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;
+ try {
+ configuration = federationFacade.getPolicyConfiguration(copyQueue);
+ } catch (YarnException e) {
+ LOG.warn("Cannot retrieve policy configured for the queue: {}, " +
+ "falling back to defaults.", copyQueue, e);
+ }
+ }
+
+ // the fallback is not configure via store, but via XML, using
+ // previously loaded configuration.
+ if (configuration == null) {
+ configuration = cachedConfiguration.get(YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY);
+ }
+
+ // if the configuration has changed since last loaded, reinit the policy
+ // based on current configuration
+ if (!cachedConfiguration.containsKey(copyQueue)
+ || !cachedConfiguration.get(copyQueue).equals(configuration)) {
Review Comment:
I would extract to make it more readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1201973872
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 16s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 13s | | trunk passed |
| +1 :green_heart: | compile | 4m 48s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 4m 32s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 9s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 5s | | the patch passed |
| +1 :green_heart: | compile | 4m 28s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 17s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) |
| +1 :green_heart: | mvnsite | 1m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 54s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 3m 23s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 140m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 85eab166f154 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1ce3194a76ee233c13e62e5dc15faf929b8ce21c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/5/testReport/ |
| Max. process+thread count | 1734 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933793274
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/LocalityRouterPolicy.java:
##########
@@ -128,53 +126,55 @@ public SubClusterId getHomeSubcluster(
ResourceRequest nodeRequest = null;
ResourceRequest rackRequest = null;
ResourceRequest anyRequest = null;
+
for (ResourceRequest rr : rrList) {
// Handle "node" requests
try {
targetId = resolver.getSubClusterForNode(rr.getResourceName());
nodeRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve node : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve node.");
}
// Handle "rack" requests
try {
resolver.getSubClustersForRack(rr.getResourceName());
rackRequest = rr;
} catch (YarnException e) {
- LOG.error("Cannot resolve rack : {}", e.getLocalizedMessage());
+ LOG.error("Cannot resolve rack.");
}
// Handle "ANY" requests
if (ResourceRequest.isAnyLocation(rr.getResourceName())) {
anyRequest = rr;
continue;
}
}
+
if (nodeRequest == null) {
- throw new YarnException("Missing node request");
+ throw new YarnException("Missing node request.");
}
if (rackRequest == null) {
- throw new YarnException("Missing rack request");
+ throw new YarnException("Missing rack request.");
}
if (anyRequest == null) {
- throw new YarnException("Missing any request");
+ throw new YarnException("Missing any request.");
}
- LOG.info(
- "Node request: " + nodeRequest.getResourceName() + ", Rack request: "
- + rackRequest.getResourceName() + ", Any request: " + anyRequest
- .getResourceName());
+
+ LOG.info("Node request: {} , Rack request: {} , Any request: {}.",
+ nodeRequest.getResourceName(), rackRequest.getResourceName(),
+ anyRequest.getResourceName());
+
// Handle "node" requests
if (validSubClusters.contains(targetId) && enabledSCs
.contains(targetId)) {
- LOG.info("Node {} is in SubCluster: {}", nodeRequest.getResourceName(),
- targetId);
+ LOG.info("Node {} is in SubCluster: {}.", nodeRequest.getResourceName(), targetId);
return targetId;
} else {
throw new YarnException("The node " + nodeRequest.getResourceName()
+ " is in a blacklist SubCluster or not active. ");
}
} catch (YarnException e) {
LOG.error("Validating resource requests failed, Falling back to "
- + "WeightedRandomRouterPolicy placement: " + e.getMessage());
+ + "WeightedRandomRouterPolicy placement.", e);
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1203970004
@goiri Please help to review the code again, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1204346815
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 12s | | trunk passed |
| +1 :green_heart: | compile | 4m 12s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 27s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 3m 57s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 57s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 4m 13s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 5m 4s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 8s | | The patch does not generate ASF License warnings. |
| | | 139m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a6a1cd88ae9b 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6dd7d058bb364209c976f2c34e9a14a065203f80 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/11/testReport/ |
| Max. process+thread count | 1717 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1204559936
@goiri Please help to review the code again, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1201642818
@goiri Please help to review the code again, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r935001976
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/BaseRouterPoliciesTest.java:
##########
@@ -36,6 +37,7 @@
import org.junit.Assert;
import org.junit.Test;
+
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1198436909
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 7s | | trunk passed |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 24 new + 1 unchanged - 0 fixed = 25 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 46s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 105m 49s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ef53c8d79070 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aeea18bd8761e31880e9937fcdc8454da47debf8 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/1/testReport/ |
| Max. process+thread count | 577 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1199151586
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 22 new + 1 unchanged - 0 fixed = 23 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 56s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 106m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ae276773f82e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 63c36964929134aa71311319ead7b5378bc5504d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/2/testReport/ |
| Max. process+thread count | 608 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1200165945
@goiri Please help to review the code again, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933035583
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/UniformRandomRouterPolicy.java:
##########
@@ -59,46 +60,13 @@ public void reinitialize(FederationPolicyInitializationContext policyContext)
setPolicyContext(policyContext);
}
- /**
- * Simply picks a random active subCluster to start the AM (this does NOT
- * depend on the weights in the policy).
- *
- * @param appSubmissionContext the {@link ApplicationSubmissionContext} that
- * has to be routed to an appropriate subCluster for execution.
- *
- * @param blackListSubClusters the list of subClusters as identified by
- * {@link SubClusterId} to blackList from the selection of the home
- * subCluster.
- *
- * @return a randomly chosen subcluster.
- *
- * @throws YarnException if there are no active subclusters.
- */
@Override
- public SubClusterId getHomeSubcluster(
- ApplicationSubmissionContext appSubmissionContext,
- List<SubClusterId> blackListSubClusters) throws YarnException {
-
- // null checks and default-queue behavior
- validate(appSubmissionContext);
-
- Map<SubClusterId, SubClusterInfo> activeSubclusters =
- getActiveSubclusters();
-
- List<SubClusterId> list = new ArrayList<>(activeSubclusters.keySet());
-
- FederationPolicyUtils.validateSubClusterAvailability(list,
- blackListSubClusters);
-
- if (blackListSubClusters != null) {
-
- // Remove from the active SubClusters from StateStore the blacklisted ones
- for (SubClusterId scId : blackListSubClusters) {
- list.remove(scId);
- }
+ protected SubClusterId chooseSubCluster(
+ String queue, Map<SubClusterId, SubClusterInfo> preSelectSubClusters) throws YarnException {
+ if(preSelectSubClusters == null || preSelectSubClusters.size() == 0) {
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1202632333
@goiri Please help to review the code again, thank you very much! I will follow up with YARN-11236.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r936082879
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestWeightedRandomRouterPolicy.java:
##########
@@ -68,10 +67,12 @@ public void configureWeights(float numSubClusters) {
SubClusterIdInfo sc = new SubClusterIdInfo("sc" + i);
// with 5% omit a subcluster
if (getRand().nextFloat() < 0.95f) {
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
- getActiveSubclusters().put(sc.toId(), sci);
+ long now = Time.now();
+ SubClusterInfo federationSubClusterInfo =
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1203732050
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 16s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 18s | | trunk passed |
| +1 :green_heart: | compile | 4m 9s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 26s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 3m 57s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 57s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 49s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 3m 26s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 131m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e0ebacc2a747 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 874c9c57933ebd4b42b3a278f92c6b1a6ae8a4c9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/10/testReport/ |
| Max. process+thread count | 1847 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri merged pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
goiri merged PR #4656:
URL: https://github.com/apache/hadoop/pull/4656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1202976744
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 21s | | trunk passed |
| +1 :green_heart: | compile | 4m 10s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 27s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 11s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 3m 57s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 57s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 9s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 48s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 3m 22s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 133m 39s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 80dbcb5b3737 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 10ddb9fd548643511f8ec90a64d58c50201d0b75 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/7/testReport/ |
| Max. process+thread count | 1869 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r936080553
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/BaseFederationPoliciesTest.java:
##########
@@ -177,11 +185,64 @@ public void setHomeSubCluster(SubClusterId homeSubCluster) {
public void setMockActiveSubclusters(int numSubclusters) {
for (int i = 1; i <= numSubclusters; i++) {
SubClusterIdInfo sc = new SubClusterIdInfo("sc" + i);
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
+ SubClusterInfo sci = SubClusterInfo.newInstance(sc.toId(),
+ "dns1:80", "dns1:81", "dns1:82", "dns1:83", SubClusterState.SC_RUNNING,
+ System.currentTimeMillis(), "something");
getActiveSubclusters().put(sc.toId(), sci);
}
}
+ public String generateClusterMetricsInfo(int id) {
+ long mem = 1024 * getRand().nextInt(277 * 100 - 1);
+ // plant a best cluster
+ if (id == 5) {
+ mem = 1024 * 277 * 100;
+ }
+ String clusterMetrics =
Review Comment:
I will fix it.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestPriorityRouterPolicy.java:
##########
@@ -54,10 +55,11 @@ public void setUp() throws Exception {
// with 5% omit a subcluster
if (getRand().nextFloat() < 0.95f || i == 5) {
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
- getActiveSubclusters().put(sc.toId(), sci);
+ long now = Time.now();
+ SubClusterInfo federationSubClusterInfo =
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r936083531
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationPoliciesTestUtil.java:
##########
@@ -159,6 +159,50 @@ public static void initializePolicyContext(
new Configuration());
}
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters,
+ FederationStateStoreFacade facade) throws YarnException {
+ FederationPolicyInitializationContext context =
+ new FederationPolicyInitializationContext(null, initResolver(), facade,
+ SubClusterId.newInstance("homesubcluster"));
+ return initializePolicyContext2(context, policy, policyInfo, activeSubClusters);
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ return initializePolicyContext2(policy, policyInfo, activeSubClusters, initFacade());
+ }
+
+ public static FederationPolicyInitializationContext initializePolicyContext2(
+ FederationPolicyInitializationContext fpc,
+ ConfigurableFederationPolicy policy, WeightedPolicyInfo policyInfo,
+ Map<SubClusterId, SubClusterInfo> activeSubClusters)
+ throws YarnException {
+ ByteBuffer buf = policyInfo.toByteBuffer();
+ fpc.setSubClusterPolicyConfiguration(SubClusterPolicyConfiguration
+ .newInstance("queue1", policy.getClass().getCanonicalName(), buf));
+
+ if (fpc.getFederationStateStoreFacade() == null) {
+ FederationStateStoreFacade facade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = mock(FederationStateStore.class);
+
+ if (activeSubClusters == null) {
+ activeSubClusters = new HashMap<>();
+ }
+ GetSubClustersInfoResponse response =
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1204643997
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 16s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 41s | | trunk passed |
| +1 :green_heart: | compile | 4m 10s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 28s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 5s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 4m 2s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 2s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 33s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 48s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 3m 18s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 133m 39s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4656 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 555de4ca9303 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 371801d5660d9b2768093f771d2f9f21caaf5e9c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/12/testReport/ |
| Max. process+thread count | 1715 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4656/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r933043370
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/WeightedRandomRouterPolicy.java:
##########
@@ -73,9 +58,8 @@ public SubClusterId getHomeSubcluster(
int pickedIndex = FederationPolicyUtils.getWeightedRandom(weightList);
if (pickedIndex == -1) {
throw new FederationPolicyException(
- "No positive weight found on active subclusters");
+ "No positive weight found on active subClusters.");
Review Comment:
I will fix it.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/BaseFederationPoliciesTest.java:
##########
@@ -28,20 +28,21 @@
import java.util.Map;
import java.util.Random;
+import org.apache.hadoop.yarn.api.protocolrecords.ReservationSubmissionRequest;
import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.server.federation.policies.amrmproxy.FederationAMRMProxyPolicy;
import org.apache.hadoop.yarn.server.federation.policies.dao.WeightedPolicyInfo;
import org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyException;
import org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyInitializationException;
import org.apache.hadoop.yarn.server.federation.policies.router.FederationRouterPolicy;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
-import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.*;
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#issuecomment-1198343021
@goiri This is the first pr after YARN-5871 Split. Please help to review the code, thank you very much!
In YARN-11235, I uploaded a design document and described the changed information in 2.1 Route Policy. The document still needs to be improved. I will improve it as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4656: YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4656:
URL: https://github.com/apache/hadoop/pull/4656#discussion_r935005118
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestLoadBasedRouterPolicy.java:
##########
@@ -131,7 +133,7 @@ public void testIfNoSubclustersWithWeightOne() {
fail();
} catch (YarnException ex) {
Assert.assertTrue(
Review Comment:
Thanks for your suggestion, I will modify the code.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestPriorityRouterPolicy.java:
##########
@@ -54,10 +54,11 @@ public void setUp() throws Exception {
// with 5% omit a subcluster
if (getRand().nextFloat() < 0.95f || i == 5) {
- SubClusterInfo sci = mock(SubClusterInfo.class);
- when(sci.getState()).thenReturn(SubClusterState.SC_RUNNING);
- when(sci.getSubClusterId()).thenReturn(sc.toId());
- getActiveSubclusters().put(sc.toId(), sci);
+ long now = System.currentTimeMillis();
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org