You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/06/06 03:16:04 UTC
[GitHub] [hadoop] zhengchenyu opened a new pull request, #4314: YARN-11153. Make proxy server support yarn federation.
zhengchenyu opened a new pull request, #4314:
URL: https://github.com/apache/hadoop/pull/4314
### Description of PR
Can't get running appmaster's web app in federation mode. (Detail message seen: https://issues.apache.org/jira/browse/YARN-10775)
Two step to solve this problem :
(a) make proxy server support federation.
(b) make router support proxy server.
It is the first step(a), detail message see: https://issues.apache.org/jira/browse/YARN-11153
### How was this patch tested?
manually test in our cluster and unit test.
### For code changes:
make proxyserver support federation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1213025113
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 14s | | trunk passed |
| +1 :green_heart: | compile | 4m 8s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 47s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 30s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 55s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | | the patch passed |
| +1 :green_heart: | compile | 3m 49s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 49s | | the patch passed |
| +1 :green_heart: | compile | 3m 19s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 13s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/18/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2 new + 105 unchanged - 4 fixed = 107 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 37s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 102m 52s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 237m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 28d010b9761e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / adaa8bb5b521c26e3b557f12417004efb7d53190 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/18/testReport/ |
| Max. process+thread count | 977 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r917164395
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertEquals(e.getMessage(), appNotFoundExceptionMsg);
Review Comment:
Isn't there a LambdaTestUtils for this?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
Review Comment:
The indentation looks weird:
```
private SubClusterInfo clusterInfo2 = SubClusterInfo.newInstance(
subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
"10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
```
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
Review Comment:
One line, the style check is 100 now.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -19,89 +19,62 @@
package org.apache.hadoop.yarn.server.webproxy;
import java.io.IOException;
+import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.ipc.RPC;
-import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
-import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
-import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
- private final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
+ private final RecordFactory recordFactory = RecordFactoryProvider
+ .getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+ this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ this.ahsAppPageUrlBase =
Review Comment:
```
this.ahsAppPageUrlBase = StringHelper.pjoin(
WebAppUtils.getHttpSchemePrefix(conf) + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
"applicationhistory", "app");
```
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,392 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web
+ * server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger
+ * or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)),
+ "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)),
+ "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0])
+ .getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:"
+ + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http://localhost:" + mockServerPort + "/amweb/apptest");
Review Comment:
public URL(String protocol, String host, int port, String file)
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##########
@@ -672,5 +676,19 @@ private FetchedAppReport getDefaultApplicationReport(ApplicationId appId,
private FetchedAppReport getDefaultApplicationReport(ApplicationId appId) {
return getDefaultApplicationReport(appId, true);
}
+
+ @VisibleForTesting
+ public String getAhsAppPageUrlBase() {
+ return ahsAppPageUrlBase != null ? ahsAppPageUrlBase
Review Comment:
Single line.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,392 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web
+ * server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger
+ * or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)),
+ "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)),
+ "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0])
+ .getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:"
+ + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http://localhost:" + mockServerPort + "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL(
+ "http://localhost:" + mockServerPort + "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http://localhost:" + mockServerPort
+ + "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS,
+ "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1,
+ "10.0.0.1:1", "10.0.0.1:1", "10.0.0.1:1",
+ "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0, "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2,
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1",
+ "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from
+ // timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId1, YarnApplicationState.RUNNING,
+ appUrl1)));
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId2, YarnApplicationState.RUNNING,
+ appUrl2)));
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenThrow(new ApplicationNotFoundException("APP NOT FOUND"));
+
+ ApplicationHistoryProtocol historyManager = Mockito
+ .mock(ApplicationHistoryProtocol.class);
+ Mockito.when(historyManager
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId4, YarnApplicationState.FINISHED, null)));
+
+ // Initial federation store.
+ FederationStateStoreFacade facade = FederationStateStoreFacade
+ .getInstance();
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo1));
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId1, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId2, subClusterId2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId3, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId4, subClusterId2));
+
+ // Start proxy server
+ WebAppProxyServerForTest proxy = new WebAppProxyServerForTest();
+ proxy.init(conf);
+ proxy.start();
+
+ // set Mocked rm and timeline
+ int proxyPort = proxy.proxy.proxyServer.getConnectorAddress(0).getPort();
+ FedAppReportFetcher appReportFetcher = proxy.proxy.appReportFetcher;
+ appReportFetcher.registerSubCluster(subClusterInfo1, appManager1);
+ appReportFetcher.registerSubCluster(subClusterInfo2, appManager2);
+ appReportFetcher.setHistoryManager(historyManager);
+
+ HttpURLConnection conn;
+ try {
+ // App1 is running in subcluster1, and original url is registered
+ // in rm of subCluster1. So proxy server will get original url from rm by
+ // getApplicationReport. Then proxy server will fetch the webapp directly.
+ URL url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId1.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/" + appId1.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App2 is running in subcluster2, and original url is registered
+ // in rm of subCluster2. So proxy server will get original url from rm by
+ // getApplicationReport. Then proxy server will fetch the webapp directly.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId2.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/" + appId2.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App3 is accepted in subcluster1, and original url is not registered
+ // yet. So proxy server will fetch the application web from rm.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId3.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/" + appId3.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App4 is finished in subcluster2, and have removed from rm, but not
+ // removed from timeline server. So proxy server will fetch the
+ // application web from timeline server.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId4.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/" + appId4.toString(), readResponse(conn));
+ conn.disconnect();
+ } finally {
+ proxy.close();
+ }
+ }
+
+ private ApplicationReport newApplicationReport(ApplicationId appId,
+ YarnApplicationState state, String origTrackingUrl) {
+ return ApplicationReport
+ .newInstance(appId, null, "testuser", null, null, null, 0, null, state,
+ null, null, 0, 0, 0, null, null, origTrackingUrl, 0f, null, null);
+ }
+
+ private String readResponse(HttpURLConnection conn) throws IOException {
+ InputStream input = conn.getInputStream();
+ byte[] bytes = new byte[input.available()];
+ input.read(bytes);
+ return new String(bytes);
+ }
+
+ private class WebAppProxyServerForTest extends CompositeService {
+
+ private WebAppProxyForTest proxy = null;
+
+ WebAppProxyServerForTest() {
+ super(WebAppProxyServer.class.getName());
+ }
+
+ @Override
+ protected synchronized void serviceInit(Configuration conf) throws Exception {
+ proxy = new WebAppProxyForTest();
+ addService(proxy);
+ super.serviceInit(conf);
+ }
+ }
+
+ /*
+ * This servlet is used for simulate the web of AppMaster, ResourceManager,
+ * TimelineServer and so on.
+ * */
+ public static class MockWebServlet extends HttpServlet {
+
+ private String role;
+
+ public MockWebServlet(String role) {
+ this.role = role;
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp)
+ throws IOException {
+ if (req.getPathInfo() != null) {
+ resp.getWriter().write(role + req.getPathInfo());
+ }
+ resp.setStatus(HttpServletResponse.SC_OK);
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp)
+ throws IOException {
+ InputStream is = req.getInputStream();
+ OutputStream os = resp.getOutputStream();
+ int c = is.read();
+ while (c > -1) {
+ os.write(c);
+ c = is.read();
+ }
+ is.close();
+ os.close();
+ resp.setStatus(HttpServletResponse.SC_OK);
+ }
+ }
+
+ private class WebAppProxyForTest extends WebAppProxy {
+
+ private HttpServer2 proxyServer;
+ private FedAppReportFetcher appReportFetcher;
+
+ @Override
+ protected void serviceStart() throws Exception {
+ Configuration conf = getConfig();
+ String bindAddress = conf.get(YarnConfiguration.PROXY_ADDRESS);
+ bindAddress = StringUtils.split(bindAddress, ':')[0];
+ AccessControlList acl = new AccessControlList(
+ conf.get(YarnConfiguration.YARN_ADMIN_ACL,
+ YarnConfiguration.DEFAULT_YARN_ADMIN_ACL));
+ proxyServer = new HttpServer2.Builder()
+ .setName("proxy")
+ .addEndpoint(
+ URI.create(WebAppUtils.getHttpSchemePrefix(conf) + bindAddress
Review Comment:
Isn't there an API with the subparts?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/DefaultAppReportFetcher.java:
##########
@@ -0,0 +1,119 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class DefaultAppReportFetcher extends AppReportFetcher {
+
+ private final ApplicationClientProtocol applicationsManager;
+ private String rmAppPageUrlBase;
+
+ /**
+ * Create a new Connection to the RM/Application History Server
+ * to fetch Application reports.
+ * @param conf the conf to use to know where the RM is.
+ */
+ public DefaultAppReportFetcher(Configuration conf) {
+ super(conf);
+ this.rmAppPageUrlBase = StringHelper
+ .pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
+ "cluster", "app");
+ try {
+ this.applicationsManager = ClientRMProxy.createRMProxy(conf,
+ ApplicationClientProtocol.class);
+ } catch (IOException e) {
+ throw new YarnRuntimeException(e);
+ }
+ }
+
+ /**
+ * Create a direct connection to RM instead of a remote connection when
+ * the proxy is running as part of the RM. Also create a remote connection to
+ * Application History Server if it is enabled.
+ * @param conf the configuration to use
+ * @param applicationsManager what to use to get the RM reports.
+ */
+ public DefaultAppReportFetcher(Configuration conf,
+ ApplicationClientProtocol applicationsManager) {
+ super(conf);
+ this.rmAppPageUrlBase = StringHelper
Review Comment:
indentation
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid)
+ .getRight();
+
+ GetApplicationReportRequest request = getRecordFactory()
+ .newRecordInstance(GetApplicationReportRequest.class);
+ request.setApplicationId(appId);
+
+ ApplicationReport appReport;
+ FetchedAppReport fetchedAppReport;
+ try {
+ appReport = applicationsManager.
+ getApplicationReport(request).getApplicationReport();
+ fetchedAppReport = new FetchedAppReport(appReport, AppReportSource.RM);
+ } catch (ApplicationNotFoundException e) {
+ if (!isAHSEnabled()) {
+ // Just throw it as usual if historyService is not enabled.
+ throw e;
+ }
+ //Fetch the application report from AHS
+ appReport = getHistoryManager().getApplicationReport(request)
+ .getApplicationReport();
+ fetchedAppReport = new FetchedAppReport(appReport, AppReportSource.AHS);
+ }
+ return fetchedAppReport;
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ return StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(getConf())
Review Comment:
Refactor this so we don't repeat.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/DefaultAppReportFetcher.java:
##########
@@ -0,0 +1,119 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class DefaultAppReportFetcher extends AppReportFetcher {
+
+ private final ApplicationClientProtocol applicationsManager;
+ private String rmAppPageUrlBase;
+
+ /**
+ * Create a new Connection to the RM/Application History Server
+ * to fetch Application reports.
+ * @param conf the conf to use to know where the RM is.
+ */
+ public DefaultAppReportFetcher(Configuration conf) {
+ super(conf);
+ this.rmAppPageUrlBase = StringHelper
Review Comment:
```
this.rmAppPageUrlBase = StringHelper.pjoin(
WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "app");
```
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertEquals(e.getMessage(), appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ try {
+ fetcher.getApplicationReport(appId2);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertEquals(e.getMessage(), appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Assert.assertNull("HistoryManager should be null as AHS is disabled",
+ history);
+ }
+
+ @Test
+ public void testGetRmAppPageUrlBase() throws IOException, YarnException {
+ testHelper(true);
+ Assert.assertEquals(fetcher.getRmAppPageUrlBase(appId1), StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(conf) + clusterInfo1
+ .getRMWebServiceAddress(), "cluster", "app"));
+ Assert.assertEquals(fetcher.getRmAppPageUrlBase(appId2), StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(conf) + clusterInfo2
+ .getRMWebServiceAddress(), "cluster", "app"));
+ }
+
+ static class FedAppReportFetcherForTest extends FedAppReportFetcher {
+
+ FedAppReportFetcherForTest(Configuration conf) {
+ super(conf);
+ }
+
+ @Override
+ protected ApplicationHistoryProtocol getAHSProxy(Configuration conf)
+ throws IOException {
+ GetApplicationReportResponse resp = Mockito.
+ mock(GetApplicationReportResponse.class);
+ history = Mockito.mock(ApplicationHistoryProtocol.class);
+ try {
+ Mockito.when(history.getApplicationReport(Mockito
+ .any(GetApplicationReportRequest.class))).thenReturn(resp);
+ } catch (YarnException e) {
+ // TODO Auto-generated catch block
+ e.printStackTrace();
Review Comment:
Surface log or error.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,392 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web
+ * server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger
+ * or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)),
+ "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)),
+ "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0])
+ .getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:"
Review Comment:
Single line and use {}
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertEquals(e.getMessage(), appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ try {
+ fetcher.getApplicationReport(appId2);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertEquals(e.getMessage(), appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Assert.assertNull("HistoryManager should be null as AHS is disabled",
+ history);
+ }
+
+ @Test
+ public void testGetRmAppPageUrlBase() throws IOException, YarnException {
+ testHelper(true);
+ Assert.assertEquals(fetcher.getRmAppPageUrlBase(appId1), StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(conf) + clusterInfo1
Review Comment:
This code yet again
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid)
+ .getRight();
+
+ GetApplicationReportRequest request = getRecordFactory()
+ .newRecordInstance(GetApplicationReportRequest.class);
+ request.setApplicationId(appId);
+
+ ApplicationReport appReport;
+ FetchedAppReport fetchedAppReport;
+ try {
+ appReport = applicationsManager.
+ getApplicationReport(request).getApplicationReport();
+ fetchedAppReport = new FetchedAppReport(appReport, AppReportSource.RM);
+ } catch (ApplicationNotFoundException e) {
+ if (!isAHSEnabled()) {
+ // Just throw it as usual if historyService is not enabled.
+ throw e;
+ }
+ //Fetch the application report from AHS
+ appReport = getHistoryManager().getApplicationReport(request)
+ .getApplicationReport();
+ fetchedAppReport = new FetchedAppReport(appReport, AppReportSource.AHS);
+ }
+ return fetchedAppReport;
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ return StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(getConf())
+ + subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void makeSureGetClusterInfo(SubClusterId scid)
+ throws YarnException, IOException {
+ if (subClusters.get(scid) == null) {
+ SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scid);
+ Configuration subClusterConf = new Configuration(getConf());
+ FederationProxyProviderUtil.updateConfForFederation(subClusterConf,
+ subClusterInfo.getSubClusterId().toString());
+ ApplicationClientProtocol proxy =
+ ClientRMProxy
+ .createRMProxy(subClusterConf, ApplicationClientProtocol.class);
+ subClusters.put(scid, Pair.of(subClusterInfo, proxy));
+ }
+ }
+
+ public void stop() {
+ for (Pair pair : this.subClusters.values()) {
+ RPC.stopProxy(pair.getRight());
+ }
+ if (this.getHistoryManager() != null) {
+ RPC.stopProxy(this.getHistoryManager());
+ }
+ }
+
+ @VisibleForTesting
+ public void registerSubCluster(SubClusterInfo info, ApplicationClientProtocol
Review Comment:
Single line
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid)
+ .getRight();
+
+ GetApplicationReportRequest request = getRecordFactory()
+ .newRecordInstance(GetApplicationReportRequest.class);
+ request.setApplicationId(appId);
+
+ ApplicationReport appReport;
Review Comment:
This looks very similar to the other code.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##########
@@ -672,5 +676,19 @@ private FetchedAppReport getDefaultApplicationReport(ApplicationId appId,
private FetchedAppReport getDefaultApplicationReport(ApplicationId appId) {
return getDefaultApplicationReport(appId, true);
}
+
+ @VisibleForTesting
+ public String getAhsAppPageUrlBase() {
+ return ahsAppPageUrlBase != null ? ahsAppPageUrlBase
+ : super.getAhsAppPageUrlBase();
+ }
+
+ @VisibleForTesting
+ public void setAhsAppPageUrlBase(Configuration conf) {
+ this.ahsAppPageUrlBase =
+ StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
+ + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
+ "applicationhistory", "app");
Review Comment:
And almost the same
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,392 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web
+ * server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger
+ * or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)),
+ "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)),
+ "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0])
+ .getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:"
+ + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http://localhost:" + mockServerPort + "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL(
+ "http://localhost:" + mockServerPort + "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http://localhost:" + mockServerPort
+ + "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS,
+ "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1,
+ "10.0.0.1:1", "10.0.0.1:1", "10.0.0.1:1",
+ "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0, "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2,
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1",
+ "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from
+ // timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId1, YarnApplicationState.RUNNING,
+ appUrl1)));
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId2, YarnApplicationState.RUNNING,
+ appUrl2)));
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenThrow(new ApplicationNotFoundException("APP NOT FOUND"));
+
+ ApplicationHistoryProtocol historyManager = Mockito
+ .mock(ApplicationHistoryProtocol.class);
+ Mockito.when(historyManager
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId4, YarnApplicationState.FINISHED, null)));
+
+ // Initial federation store.
+ FederationStateStoreFacade facade = FederationStateStoreFacade
+ .getInstance();
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo1));
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId1, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId2, subClusterId2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId3, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId4, subClusterId2));
+
+ // Start proxy server
+ WebAppProxyServerForTest proxy = new WebAppProxyServerForTest();
+ proxy.init(conf);
+ proxy.start();
+
+ // set Mocked rm and timeline
+ int proxyPort = proxy.proxy.proxyServer.getConnectorAddress(0).getPort();
+ FedAppReportFetcher appReportFetcher = proxy.proxy.appReportFetcher;
+ appReportFetcher.registerSubCluster(subClusterInfo1, appManager1);
+ appReportFetcher.registerSubCluster(subClusterInfo2, appManager2);
+ appReportFetcher.setHistoryManager(historyManager);
+
+ HttpURLConnection conn;
Review Comment:
Why not in the try?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r921547872
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ return StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(getConf()) + subClusterInfo.getRMWebServiceAddress(),
+ "cluster", "app");
+ }
+
+ private void makeSureGetClusterInfo(SubClusterId scid) throws YarnException, IOException {
+ if (subClusters.get(scid) == null) {
Review Comment:
```
if (subClusters.contains(scid)) {
return;
}
```
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ return StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(getConf()) + subClusterInfo.getRMWebServiceAddress(),
+ "cluster", "app");
+ }
+
+ private void makeSureGetClusterInfo(SubClusterId scid) throws YarnException, IOException {
Review Comment:
scid -> scId
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ return StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(getConf()) + subClusterInfo.getRMWebServiceAddress(),
+ "cluster", "app");
+ }
+
+ private void makeSureGetClusterInfo(SubClusterId scid) throws YarnException, IOException {
Review Comment:
The name `makeSureGetClusterInfo()` is a little weird, this is more like `createSubclusterIfAbsent()`
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ makeSureGetClusterInfo(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ return StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(getConf()) + subClusterInfo.getRMWebServiceAddress(),
Review Comment:
Extract the scheme.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,172 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import static org.junit.Assert.fail;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1, subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2, subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1.getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2.getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws Exception {
+ testHelper(false);
+
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ LambdaTestUtils.intercept(ApplicationNotFoundException.class, appNotFoundExceptionMsg,
+ () -> fetcher.getApplicationReport(appId1));
+ LambdaTestUtils.intercept(ApplicationNotFoundException.class, appNotFoundExceptionMsg,
+ () -> fetcher.getApplicationReport(appId2));
+
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Assert.assertNull("HistoryManager should be null as AHS is disabled", history);
+ }
+
+ @Test
+ public void testGetRmAppPageUrlBase() throws IOException, YarnException {
+ testHelper(true);
+ Assert.assertEquals(fetcher.getRmAppPageUrlBase(appId1), StringHelper.pjoin(
+ WebAppUtils.getHttpSchemePrefix(conf) + clusterInfo1.getRMWebServiceAddress(), "cluster",
Review Comment:
Extracting the scheme would make it cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r920702161
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
Review Comment:
Oh, I use 80 in my idea all the time. I will reconstruct all code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r889779662
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ Configuration conf;
+ static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ try {
+ fetcher.getApplicationReport(appId2);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ if (history != null) {
Review Comment:
I will correct it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
URL: https://github.com/apache/hadoop/pull/4314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r920717298
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,392 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web
+ * server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger
+ * or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)),
+ "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)),
+ "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0])
+ .getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:"
+ + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http://localhost:" + mockServerPort + "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL(
+ "http://localhost:" + mockServerPort + "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http://localhost:" + mockServerPort
+ + "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS,
+ "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1,
+ "10.0.0.1:1", "10.0.0.1:1", "10.0.0.1:1",
+ "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0, "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2,
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1",
+ "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from
+ // timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId1, YarnApplicationState.RUNNING,
+ appUrl1)));
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId2, YarnApplicationState.RUNNING,
+ appUrl2)));
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenThrow(new ApplicationNotFoundException("APP NOT FOUND"));
+
+ ApplicationHistoryProtocol historyManager = Mockito
+ .mock(ApplicationHistoryProtocol.class);
+ Mockito.when(historyManager
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId4, YarnApplicationState.FINISHED, null)));
+
+ // Initial federation store.
+ FederationStateStoreFacade facade = FederationStateStoreFacade
+ .getInstance();
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo1));
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId1, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId2, subClusterId2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId3, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId4, subClusterId2));
+
+ // Start proxy server
+ WebAppProxyServerForTest proxy = new WebAppProxyServerForTest();
+ proxy.init(conf);
+ proxy.start();
+
+ // set Mocked rm and timeline
+ int proxyPort = proxy.proxy.proxyServer.getConnectorAddress(0).getPort();
+ FedAppReportFetcher appReportFetcher = proxy.proxy.appReportFetcher;
+ appReportFetcher.registerSubCluster(subClusterInfo1, appManager1);
+ appReportFetcher.registerSubCluster(subClusterInfo2, appManager2);
+ appReportFetcher.setHistoryManager(historyManager);
+
+ HttpURLConnection conn;
+ try {
+ // App1 is running in subcluster1, and original url is registered
+ // in rm of subCluster1. So proxy server will get original url from rm by
+ // getApplicationReport. Then proxy server will fetch the webapp directly.
+ URL url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId1.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/" + appId1.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App2 is running in subcluster2, and original url is registered
+ // in rm of subCluster2. So proxy server will get original url from rm by
+ // getApplicationReport. Then proxy server will fetch the webapp directly.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId2.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/" + appId2.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App3 is accepted in subcluster1, and original url is not registered
+ // yet. So proxy server will fetch the application web from rm.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId3.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/" + appId3.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App4 is finished in subcluster2, and have removed from rm, but not
+ // removed from timeline server. So proxy server will fetch the
+ // application web from timeline server.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId4.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/" + appId4.toString(), readResponse(conn));
+ conn.disconnect();
+ } finally {
+ proxy.close();
+ }
+ }
+
+ private ApplicationReport newApplicationReport(ApplicationId appId,
+ YarnApplicationState state, String origTrackingUrl) {
+ return ApplicationReport
+ .newInstance(appId, null, "testuser", null, null, null, 0, null, state,
+ null, null, 0, 0, 0, null, null, origTrackingUrl, 0f, null, null);
+ }
+
+ private String readResponse(HttpURLConnection conn) throws IOException {
+ InputStream input = conn.getInputStream();
+ byte[] bytes = new byte[input.available()];
+ input.read(bytes);
+ return new String(bytes);
+ }
+
+ private class WebAppProxyServerForTest extends CompositeService {
+
+ private WebAppProxyForTest proxy = null;
+
+ WebAppProxyServerForTest() {
+ super(WebAppProxyServer.class.getName());
+ }
+
+ @Override
+ protected synchronized void serviceInit(Configuration conf) throws Exception {
+ proxy = new WebAppProxyForTest();
+ addService(proxy);
+ super.serviceInit(conf);
+ }
+ }
+
+ /*
+ * This servlet is used for simulate the web of AppMaster, ResourceManager,
+ * TimelineServer and so on.
+ * */
+ public static class MockWebServlet extends HttpServlet {
+
+ private String role;
+
+ public MockWebServlet(String role) {
+ this.role = role;
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp)
+ throws IOException {
+ if (req.getPathInfo() != null) {
+ resp.getWriter().write(role + req.getPathInfo());
+ }
+ resp.setStatus(HttpServletResponse.SC_OK);
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp)
+ throws IOException {
+ InputStream is = req.getInputStream();
+ OutputStream os = resp.getOutputStream();
+ int c = is.read();
+ while (c > -1) {
+ os.write(c);
+ c = is.read();
+ }
+ is.close();
+ os.close();
+ resp.setStatus(HttpServletResponse.SC_OK);
+ }
+ }
+
+ private class WebAppProxyForTest extends WebAppProxy {
+
+ private HttpServer2 proxyServer;
+ private FedAppReportFetcher appReportFetcher;
+
+ @Override
+ protected void serviceStart() throws Exception {
+ Configuration conf = getConfig();
+ String bindAddress = conf.get(YarnConfiguration.PROXY_ADDRESS);
+ bindAddress = StringUtils.split(bindAddress, ':')[0];
+ AccessControlList acl = new AccessControlList(
+ conf.get(YarnConfiguration.YARN_ADMIN_ACL,
+ YarnConfiguration.DEFAULT_YARN_ADMIN_ACL));
+ proxyServer = new HttpServer2.Builder()
+ .setName("proxy")
+ .addEndpoint(
+ URI.create(WebAppUtils.getHttpSchemePrefix(conf) + bindAddress
Review Comment:
> Isn't there an API with the subparts?
Yes, there are no api like URI(String scheme, String host, int port).
But there is an API URI(String scheme, String userInfo, String host, int port,String path, String query, String fragment). In this construct method, firstly construct a string, then parse. I think it is not better than URL(String spec)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r920733448
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertEquals(e.getMessage(), appNotFoundExceptionMsg);
Review Comment:
I will change to LambdaTestUtils.intercept!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1185548563
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 19s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 36s | | trunk passed |
| +1 :green_heart: | compile | 4m 28s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 15s | | the patch passed |
| +1 :green_heart: | compile | 4m 17s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 17s | | the patch passed |
| +1 :green_heart: | compile | 3m 33s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 15s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/16/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2 new + 105 unchanged - 4 fixed = 107 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 59s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 7s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 102m 19s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 245m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3fbc269f0a95 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / da8d70a528e9b522d4dadbd3e24c394981136359 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/16/testReport/ |
| Max. process+thread count | 941 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1134269618
@goiri Can you help me review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945244909
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/DefaultAppReportFetcher.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class DefaultAppReportFetcher extends AppReportFetcher {
+
+ private final ApplicationClientProtocol applicationsManager;
+ private String rmAppPageUrlBase;
+
+ /**
+ * Create a new Connection to the RM/Application History Server
+ * to fetch Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public DefaultAppReportFetcher(Configuration conf) {
+ super(conf);
+ this.rmAppPageUrlBase =
+ StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf), "cluster", "app");
+ try {
+ this.applicationsManager = ClientRMProxy.createRMProxy(conf, ApplicationClientProtocol.class);
+ } catch (IOException e) {
+ throw new YarnRuntimeException(e);
+ }
+ }
+
+ /**
+ * Create a direct connection to RM instead of a remote connection when
+ * the proxy is running as part of the RM. Also create a remote connection to
+ * Application History Server if it is enabled.
+ *
+ * @param conf the configuration to use
+ * @param applicationsManager what to use to get the RM reports.
+ */
+ public DefaultAppReportFetcher(Configuration conf,
+ ApplicationClientProtocol applicationsManager) {
Review Comment:
indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1127750625
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 56s | | trunk passed |
| +1 :green_heart: | compile | 4m 5s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 34s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 52s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 52s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 12s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 11 new + 95 unchanged - 4 fixed = 106 total (was 99) |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| -1 :x: | javadoc | 0m 30s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/1/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 2 new + 9 unchanged - 0 fixed = 11 total (was 9) |
| -1 :x: | javadoc | 0m 30s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/1/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 9 unchanged - 0 fixed = 11 total (was 9) |
| +1 :green_heart: | spotbugs | 3m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 101m 23s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 223m 45s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 2d9cfae212d1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 118f0187cecc60a2e571c35dd39b081b77d92e37 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/1/testReport/ |
| Max. process+thread count | 924 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1161310665
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 51s | | trunk passed |
| +1 :green_heart: | compile | 4m 20s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 58s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 23s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 13s | | the patch passed |
| +1 :green_heart: | compile | 4m 3s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 3s | | the patch passed |
| +1 :green_heart: | compile | 3m 15s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 7s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 10 new + 107 unchanged - 2 fixed = 117 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 101m 40s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. |
| | | 238m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7069a2fadc7d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4da15f6aa54aa147ca0c800869eb04657ef5aec7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/7/testReport/ |
| Max. process+thread count | 927 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1161326655
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 2s | | trunk passed |
| +1 :green_heart: | compile | 4m 3s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 29s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 36s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 25s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 59s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | compile | 3m 19s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 7s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 10 new + 107 unchanged - 2 fixed = 117 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 4s | | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 1m 28s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/8/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 102m 18s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 238m 36s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b3641b950297 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e1bec4858141853e174390546ed520fef245331b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/8/testReport/ |
| Max. process+thread count | 958 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r928788350
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -112,51 +87,74 @@ protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
* @throws YarnException on any error.
* @throws IOException
*/
- public FetchedAppReport getApplicationReport(ApplicationId appId)
- throws YarnException, IOException {
- GetApplicationReportRequest request = recordFactory
- .newRecordInstance(GetApplicationReportRequest.class);
+ public abstract FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException;
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param applicationsManager what to use to get the RM reports.
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ protected FetchedAppReport getApplicationReport(ApplicationClientProtocol applicationsManager,
+ ApplicationId appId)
Review Comment:
indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r934775212
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+ this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ String historyUrl = WebAppUtils.getAHSWebAppURLWithoutScheme(conf);
+ this.ahsAppPageUrlBase = StringHelper.pjoin(scheme + historyUrl, "applicationhistory", "app");
}
- this.conf = conf;
try {
- applicationsManager = ClientRMProxy.createRMProxy(conf,
- ApplicationClientProtocol.class);
- if (isAHSEnabled) {
- historyManager = getAHSProxy(conf);
+ if (this.isAHSEnabled) {
+ this.historyManager = getAHSProxy(conf);
} else {
this.historyManager = null;
}
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
-
- /**
- * Create a direct connection to RM instead of a remote connection when
- * the proxy is running as part of the RM. Also create a remote connection to
- * Application History Server if it is enabled.
- * @param conf the configuration to use
- * @param applicationsManager what to use to get the RM reports.
- */
- public AppReportFetcher(Configuration conf, ApplicationClientProtocol applicationsManager) {
- if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
- }
- this.conf = conf;
- this.applicationsManager = applicationsManager;
- if (isAHSEnabled) {
- try {
- historyManager = getAHSProxy(conf);
- } catch (IOException e) {
- throw new YarnRuntimeException(e);
- }
- } else {
- this.historyManager = null;
- }
- }
protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
throws IOException {
- return AHSProxy.createAHSProxy(configuration,
- ApplicationHistoryProtocol.class,
- configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
+ return AHSProxy.createAHSProxy(configuration, ApplicationHistoryProtocol.class,
+ configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
Review Comment:
I think the indentation is correct in both cases.
TBH, I think what would help here is to extract,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945385056
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreFacade.java:
##########
@@ -610,7 +610,6 @@ public boolean equals(Object obj) {
TResult invoke(T input) throws Exception;
}
-
Review Comment:
Avoid this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1185267713
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 44s | | trunk passed |
| +1 :green_heart: | compile | 4m 9s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 26s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 56s | | the patch passed |
| +1 :green_heart: | compile | 3m 57s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 57s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 13s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/15/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 11 new + 105 unchanged - 4 fixed = 116 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 9s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 48s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 22s | | hadoop-yarn-server-web-proxy in the patch passed. |
| -1 :x: | unit | 103m 45s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/15/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 242m 30s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux db4752114a54 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 50f311870d32ee137eff9386ec93e37338c31250 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/15/testReport/ |
| Max. process+thread count | 934 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1184194925
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 29m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 54s | | trunk passed |
| +1 :green_heart: | compile | 4m 3s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 30s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 12s | | the patch passed |
| +1 :green_heart: | compile | 4m 22s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 22s | | the patch passed |
| +1 :green_heart: | compile | 3m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 20s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/12/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 106 unchanged - 3 fixed = 107 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 53s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 7s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 102m 11s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 256m 51s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fe9fd854a1a5 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 587d869c139d3c453f19714cb4afbe20e4b9b9bd |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/12/testReport/ |
| Max. process+thread count | 957 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1213071121
@goiri How about merge to trunk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r928791799
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)), "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)), "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0]).getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:" + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http", "localhost", mockServerPort, "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL("http", "localhost", mockServerPort, "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http", "localhost", mockServerPort, "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS, "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1",
+ "10.0.0.1:1", "10.0.0.1:1", "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0,
+ "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1",
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId1, YarnApplicationState.RUNNING, appUrl1)));
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2.getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId2, YarnApplicationState.RUNNING, appUrl2)));
+ Mockito.when(appManager2.getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenThrow(new ApplicationNotFoundException("APP NOT FOUND"));
+
+ ApplicationHistoryProtocol historyManager = Mockito.mock(ApplicationHistoryProtocol.class);
+ Mockito
+ .when(historyManager.getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId4, YarnApplicationState.FINISHED, null)));
Review Comment:
indentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1147122398
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 13s | | trunk passed |
| +1 :green_heart: | compile | 4m 36s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 30s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 16s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 3m 57s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 57s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 12s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 12 new + 94 unchanged - 4 fixed = 106 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 35s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | | hadoop-yarn-server-web-proxy in the patch passed. |
| -1 :x: | unit | 101m 24s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 231m 54s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 78794097048d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fe396f6762f2b2d558012afd9a5104db0fd8522e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/5/testReport/ |
| Max. process+thread count | 935 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669146353
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 34s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 29s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | spotbugs | 0m 43s | [/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in trunk failed. |
| +1 :green_heart: | shadedclient | 23m 46s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 4s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 1m 43s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 1m 31s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 43s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 59s | | hadoop-yarn-server-web-proxy in the patch passed. |
| -1 :x: | unit | 86m 19s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 202m 37s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 410c096cdd46 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c389da12f38bceb5af76865f9508bd35457f6403 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669555702
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 26s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 13s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 31s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 1m 40s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 40s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 0s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 85m 58s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not generate ASF License warnings. |
| | | 190m 24s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/28/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 89dae1d090e4 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a16b568ac37c0acd36c80f2628f1e7b85017cfa7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/28/testReport/ |
| Max. process+thread count | 929 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/28/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r928791799
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)), "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)), "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0]).getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:" + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http", "localhost", mockServerPort, "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL("http", "localhost", mockServerPort, "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http", "localhost", mockServerPort, "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS, "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1",
+ "10.0.0.1:1", "10.0.0.1:1", "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0,
+ "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1",
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId1, YarnApplicationState.RUNNING, appUrl1)));
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2.getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId2, YarnApplicationState.RUNNING, appUrl2)));
+ Mockito.when(appManager2.getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenThrow(new ApplicationNotFoundException("APP NOT FOUND"));
+
+ ApplicationHistoryProtocol historyManager = Mockito.mock(ApplicationHistoryProtocol.class);
+ Mockito
+ .when(historyManager.getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId4, YarnApplicationState.FINISHED, null)));
Review Comment:
indentation
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)), "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)), "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0]).getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:" + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http", "localhost", mockServerPort, "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL("http", "localhost", mockServerPort, "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http", "localhost", mockServerPort, "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS, "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1",
+ "10.0.0.1:1", "10.0.0.1:1", "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0,
+ "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1",
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId1, YarnApplicationState.RUNNING, appUrl1)));
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2.getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId2, YarnApplicationState.RUNNING, appUrl2)));
Review Comment:
indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945120827
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##########
@@ -361,8 +362,8 @@ public void testAppReportForEmptyTrackingUrl() throws Exception {
proxyConn.getURL().toString().equals(appAddressInRm));
//set AHS_ENBALED = true to simulate getting the app report from AHS
- configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- true);
+ configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ proxy.proxy.appReportFetcher.setAhsAppPageUrlBase(configuration);
Review Comment:
Because the code style of original code is wrong, I will fix all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214309067
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 47s | | trunk passed |
| +1 :green_heart: | compile | 4m 5s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 38s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 24s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 51s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 25s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvninstall | 0m 40s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 2m 3s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 2m 3s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 51s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 51s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 11s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 84 unchanged - 24 fixed = 84 total (was 108) |
| -1 :x: | mvnsite | 0m 27s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvnsite | 0m 42s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | javadoc | 0m 36s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javadoc | 0m 46s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 343 unchanged - 0 fixed = 344 total (was 343) |
| -1 :x: | spotbugs | 0m 28s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | spotbugs | 0m 39s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | shadedclient | 9m 15s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 3s | | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 0m 32s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | unit | 0m 44s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. |
| | | 119m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c1c6eb7b73a9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e78d8d3e366aa55deb1044733eb607ce89104794 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945404189
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##########
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
- this.rmAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
- this.ahsAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException, IOException {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This code should show like this. If refactor is only for clean code. I think it is not very clean.
```
ServletContext context = getServletContext();
AppReportFetcher report =
((AppReportFetcher) context.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r950851516
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+ return StringHelper.pjoin(scheme + subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws YarnException, IOException {
+ if (subClusters.containsKey(scId)) {
+ return;
+ }
+ SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+ Configuration subClusterConf = new Configuration(getConf());
+ FederationProxyProviderUtil
+ .updateConfForFederation(subClusterConf, subClusterInfo.getSubClusterId().toString());
Review Comment:
Is toString necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214385755
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 34s | | trunk passed |
| +1 :green_heart: | compile | 4m 8s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 36s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 37s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 55s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 23s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 3m 53s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 53s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 14s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 84 unchanged - 24 fixed = 84 total (was 108) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 103m 0s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 238m 44s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/22/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6503ca331e17 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 22d57d78a676cfdfbb10d220fa50ac9c85ad5fa6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/22/testReport/ |
| Max. process+thread count | 922 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/22/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1201052179
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 32s | | trunk passed |
| +1 :green_heart: | compile | 4m 4s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 37s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 43s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 7s | | the patch passed |
| +1 :green_heart: | compile | 3m 51s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 51s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 13s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/17/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2 new + 105 unchanged - 4 fixed = 107 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 56s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 35s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 103m 0s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 238m 46s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux db800ea21999 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 14a168a3362c32e8f782a4389e031c6b81083305 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/17/testReport/ |
| Max. process+thread count | 930 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1184203770
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 36m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 4s | | trunk passed |
| +1 :green_heart: | compile | 4m 24s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 41s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 4m 15s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 15s | | the patch passed |
| +1 :green_heart: | compile | 3m 26s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 17s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/11/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 106 unchanged - 3 fixed = 107 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 3s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 56s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 10s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 24s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 106m 55s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 271m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 03d8b4c4c814 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a1353d0692ab3139c9eab19987036a93a2fdade0 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/11/testReport/ |
| Max. process+thread count | 956 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r920717298
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,392 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web
+ * server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger
+ * or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)),
+ "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)),
+ "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0])
+ .getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:"
+ + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http://localhost:" + mockServerPort + "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL(
+ "http://localhost:" + mockServerPort + "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http://localhost:" + mockServerPort
+ + "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS,
+ "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1,
+ "10.0.0.1:1", "10.0.0.1:1", "10.0.0.1:1",
+ "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0, "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2,
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1",
+ "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from
+ // timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId1, YarnApplicationState.RUNNING,
+ appUrl1)));
+ Mockito.when(appManager1
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito
+ .mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId2, YarnApplicationState.RUNNING,
+ appUrl2)));
+ Mockito.when(appManager2
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenThrow(new ApplicationNotFoundException("APP NOT FOUND"));
+
+ ApplicationHistoryProtocol historyManager = Mockito
+ .mock(ApplicationHistoryProtocol.class);
+ Mockito.when(historyManager
+ .getApplicationReport(GetApplicationReportRequest.newInstance(appId4)))
+ .thenReturn(GetApplicationReportResponse.newInstance(
+ newApplicationReport(appId4, YarnApplicationState.FINISHED, null)));
+
+ // Initial federation store.
+ FederationStateStoreFacade facade = FederationStateStoreFacade
+ .getInstance();
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo1));
+ facade.getStateStore().registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId1, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId2, subClusterId2));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId3, subClusterId1));
+ facade.addApplicationHomeSubCluster(
+ ApplicationHomeSubCluster.newInstance(appId4, subClusterId2));
+
+ // Start proxy server
+ WebAppProxyServerForTest proxy = new WebAppProxyServerForTest();
+ proxy.init(conf);
+ proxy.start();
+
+ // set Mocked rm and timeline
+ int proxyPort = proxy.proxy.proxyServer.getConnectorAddress(0).getPort();
+ FedAppReportFetcher appReportFetcher = proxy.proxy.appReportFetcher;
+ appReportFetcher.registerSubCluster(subClusterInfo1, appManager1);
+ appReportFetcher.registerSubCluster(subClusterInfo2, appManager2);
+ appReportFetcher.setHistoryManager(historyManager);
+
+ HttpURLConnection conn;
+ try {
+ // App1 is running in subcluster1, and original url is registered
+ // in rm of subCluster1. So proxy server will get original url from rm by
+ // getApplicationReport. Then proxy server will fetch the webapp directly.
+ URL url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId1.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/" + appId1.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App2 is running in subcluster2, and original url is registered
+ // in rm of subCluster2. So proxy server will get original url from rm by
+ // getApplicationReport. Then proxy server will fetch the webapp directly.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId2.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/" + appId2.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App3 is accepted in subcluster1, and original url is not registered
+ // yet. So proxy server will fetch the application web from rm.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId3.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/" + appId3.toString(), readResponse(conn));
+ conn.disconnect();
+
+ // App4 is finished in subcluster2, and have removed from rm, but not
+ // removed from timeline server. So proxy server will fetch the
+ // application web from timeline server.
+ url = new URL("http://localhost:" + proxyPort + "/proxy/" + appId4.toString());
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/" + appId4.toString(), readResponse(conn));
+ conn.disconnect();
+ } finally {
+ proxy.close();
+ }
+ }
+
+ private ApplicationReport newApplicationReport(ApplicationId appId,
+ YarnApplicationState state, String origTrackingUrl) {
+ return ApplicationReport
+ .newInstance(appId, null, "testuser", null, null, null, 0, null, state,
+ null, null, 0, 0, 0, null, null, origTrackingUrl, 0f, null, null);
+ }
+
+ private String readResponse(HttpURLConnection conn) throws IOException {
+ InputStream input = conn.getInputStream();
+ byte[] bytes = new byte[input.available()];
+ input.read(bytes);
+ return new String(bytes);
+ }
+
+ private class WebAppProxyServerForTest extends CompositeService {
+
+ private WebAppProxyForTest proxy = null;
+
+ WebAppProxyServerForTest() {
+ super(WebAppProxyServer.class.getName());
+ }
+
+ @Override
+ protected synchronized void serviceInit(Configuration conf) throws Exception {
+ proxy = new WebAppProxyForTest();
+ addService(proxy);
+ super.serviceInit(conf);
+ }
+ }
+
+ /*
+ * This servlet is used for simulate the web of AppMaster, ResourceManager,
+ * TimelineServer and so on.
+ * */
+ public static class MockWebServlet extends HttpServlet {
+
+ private String role;
+
+ public MockWebServlet(String role) {
+ this.role = role;
+ }
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp)
+ throws IOException {
+ if (req.getPathInfo() != null) {
+ resp.getWriter().write(role + req.getPathInfo());
+ }
+ resp.setStatus(HttpServletResponse.SC_OK);
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp)
+ throws IOException {
+ InputStream is = req.getInputStream();
+ OutputStream os = resp.getOutputStream();
+ int c = is.read();
+ while (c > -1) {
+ os.write(c);
+ c = is.read();
+ }
+ is.close();
+ os.close();
+ resp.setStatus(HttpServletResponse.SC_OK);
+ }
+ }
+
+ private class WebAppProxyForTest extends WebAppProxy {
+
+ private HttpServer2 proxyServer;
+ private FedAppReportFetcher appReportFetcher;
+
+ @Override
+ protected void serviceStart() throws Exception {
+ Configuration conf = getConfig();
+ String bindAddress = conf.get(YarnConfiguration.PROXY_ADDRESS);
+ bindAddress = StringUtils.split(bindAddress, ':')[0];
+ AccessControlList acl = new AccessControlList(
+ conf.get(YarnConfiguration.YARN_ADMIN_ACL,
+ YarnConfiguration.DEFAULT_YARN_ADMIN_ACL));
+ proxyServer = new HttpServer2.Builder()
+ .setName("proxy")
+ .addEndpoint(
+ URI.create(WebAppUtils.getHttpSchemePrefix(conf) + bindAddress
Review Comment:
> Isn't there an API with the subparts?
Yes, there are no api like URI(String scheme, String host, int port).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1147119396
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 6s | | trunk passed |
| +1 :green_heart: | compile | 4m 25s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 4m 9s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 16s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 11 new + 95 unchanged - 3 fixed = 106 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in the patch passed. |
| -1 :x: | unit | 101m 32s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 228m 42s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3584c6787078 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fe396f6762f2b2d558012afd9a5104db0fd8522e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/4/testReport/ |
| Max. process+thread count | 971 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r889779590
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ Configuration conf;
+ static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ try {
+ fetcher.getApplicationReport(appId2);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
Review Comment:
In fact, no problem here, the object checked is same object. But it is not normal way indeed. I will correct it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
URL: https://github.com/apache/hadoop/pull/4314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1227976708
Is there any problem about this PR? How about merge trunk? Then I will continue YARN-11153. @goiri @slfan1989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956840427
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+ return StringHelper.pjoin(scheme + subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws YarnException, IOException {
+ if (subClusters.containsKey(scId)) {
+ return;
+ }
+ SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+ Configuration subClusterConf = new Configuration(getConf());
+ FederationProxyProviderUtil
+ .updateConfForFederation(subClusterConf, subClusterInfo.getSubClusterId().toString());
Review Comment:
Hi, I tried to remove toString. It will cause compile error. Warns "Required type: String, Provided: SubClusterId".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r928787939
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+ this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ String historyUrl = WebAppUtils.getAHSWebAppURLWithoutScheme(conf);
+ this.ahsAppPageUrlBase = StringHelper.pjoin(scheme + historyUrl, "applicationhistory", "app");
}
- this.conf = conf;
try {
- applicationsManager = ClientRMProxy.createRMProxy(conf,
- ApplicationClientProtocol.class);
- if (isAHSEnabled) {
- historyManager = getAHSProxy(conf);
+ if (this.isAHSEnabled) {
+ this.historyManager = getAHSProxy(conf);
} else {
this.historyManager = null;
}
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
-
- /**
- * Create a direct connection to RM instead of a remote connection when
- * the proxy is running as part of the RM. Also create a remote connection to
- * Application History Server if it is enabled.
- * @param conf the configuration to use
- * @param applicationsManager what to use to get the RM reports.
- */
- public AppReportFetcher(Configuration conf, ApplicationClientProtocol applicationsManager) {
- if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
- }
- this.conf = conf;
- this.applicationsManager = applicationsManager;
- if (isAHSEnabled) {
- try {
- historyManager = getAHSProxy(conf);
- } catch (IOException e) {
- throw new YarnRuntimeException(e);
- }
- } else {
- this.historyManager = null;
- }
- }
protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
throws IOException {
- return AHSProxy.createAHSProxy(configuration,
- ApplicationHistoryProtocol.class,
- configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
+ return AHSProxy.createAHSProxy(configuration, ApplicationHistoryProtocol.class,
+ configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
Review Comment:
indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1162568830
@goiri I reconstruct my UT, add new test class TestWebAppProxyServletFed. It will simulate the proxy link in federation mode. Can you please review it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r940821669
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+ this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ String historyUrl = WebAppUtils.getAHSWebAppURLWithoutScheme(conf);
+ this.ahsAppPageUrlBase = StringHelper.pjoin(scheme + historyUrl, "applicationhistory", "app");
}
- this.conf = conf;
try {
- applicationsManager = ClientRMProxy.createRMProxy(conf,
- ApplicationClientProtocol.class);
- if (isAHSEnabled) {
- historyManager = getAHSProxy(conf);
+ if (this.isAHSEnabled) {
+ this.historyManager = getAHSProxy(conf);
} else {
this.historyManager = null;
}
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
-
- /**
- * Create a direct connection to RM instead of a remote connection when
- * the proxy is running as part of the RM. Also create a remote connection to
- * Application History Server if it is enabled.
- * @param conf the configuration to use
- * @param applicationsManager what to use to get the RM reports.
- */
- public AppReportFetcher(Configuration conf, ApplicationClientProtocol applicationsManager) {
- if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
- }
- this.conf = conf;
- this.applicationsManager = applicationsManager;
- if (isAHSEnabled) {
- try {
- historyManager = getAHSProxy(conf);
- } catch (IOException e) {
- throw new YarnRuntimeException(e);
- }
- } else {
- this.historyManager = null;
- }
- }
protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
throws IOException {
- return AHSProxy.createAHSProxy(configuration,
- ApplicationHistoryProtocol.class,
- configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
+ return AHSProxy.createAHSProxy(configuration, ApplicationHistoryProtocol.class,
+ configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
Review Comment:
> I think the indentation is correct in both cases. TBH, I think what would help here is to extract,
Should I need to extract below code to a single function?
```
configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945172390
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+ this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ String historyUrl = WebAppUtils.getAHSWebAppURLWithoutScheme(conf);
+ this.ahsAppPageUrlBase = StringHelper.pjoin(scheme + historyUrl, "applicationhistory", "app");
}
- this.conf = conf;
try {
- applicationsManager = ClientRMProxy.createRMProxy(conf,
- ApplicationClientProtocol.class);
- if (isAHSEnabled) {
- historyManager = getAHSProxy(conf);
+ if (this.isAHSEnabled) {
+ this.historyManager = getAHSProxy(conf);
} else {
this.historyManager = null;
}
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
-
- /**
- * Create a direct connection to RM instead of a remote connection when
- * the proxy is running as part of the RM. Also create a remote connection to
- * Application History Server if it is enabled.
- * @param conf the configuration to use
- * @param applicationsManager what to use to get the RM reports.
- */
- public AppReportFetcher(Configuration conf, ApplicationClientProtocol applicationsManager) {
- if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
- }
- this.conf = conf;
- this.applicationsManager = applicationsManager;
- if (isAHSEnabled) {
- try {
- historyManager = getAHSProxy(conf);
- } catch (IOException e) {
- throw new YarnRuntimeException(e);
- }
- } else {
- this.historyManager = null;
- }
- }
protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
throws IOException {
- return AHSProxy.createAHSProxy(configuration,
- ApplicationHistoryProtocol.class,
- configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
+ return AHSProxy.createAHSProxy(configuration, ApplicationHistoryProtocol.class,
+ configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
Review Comment:
I think the cleanest is to do:
```
InetSocketAddress addr = configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT)
return AHSProxy.createAHSProxy(configuration, ApplicationHistoryProtocol.class, addr);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945245040
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import static org.junit.Assert.fail;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
Review Comment:
Can these parameters be initialized in setUp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1146978859
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 19s | | https://github.com/apache/hadoop/pull/4314 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/3/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r934201179
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+ this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ String historyUrl = WebAppUtils.getAHSWebAppURLWithoutScheme(conf);
+ this.ahsAppPageUrlBase = StringHelper.pjoin(scheme + historyUrl, "applicationhistory", "app");
}
- this.conf = conf;
try {
- applicationsManager = ClientRMProxy.createRMProxy(conf,
- ApplicationClientProtocol.class);
- if (isAHSEnabled) {
- historyManager = getAHSProxy(conf);
+ if (this.isAHSEnabled) {
+ this.historyManager = getAHSProxy(conf);
} else {
this.historyManager = null;
}
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
-
- /**
- * Create a direct connection to RM instead of a remote connection when
- * the proxy is running as part of the RM. Also create a remote connection to
- * Application History Server if it is enabled.
- * @param conf the configuration to use
- * @param applicationsManager what to use to get the RM reports.
- */
- public AppReportFetcher(Configuration conf, ApplicationClientProtocol applicationsManager) {
- if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
- }
- this.conf = conf;
- this.applicationsManager = applicationsManager;
- if (isAHSEnabled) {
- try {
- historyManager = getAHSProxy(conf);
- } catch (IOException e) {
- throw new YarnRuntimeException(e);
- }
- } else {
- this.historyManager = null;
- }
- }
protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
throws IOException {
- return AHSProxy.createAHSProxy(configuration,
- ApplicationHistoryProtocol.class,
- configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
+ return AHSProxy.createAHSProxy(configuration, ApplicationHistoryProtocol.class,
+ configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
Review Comment:
Sorry, I didn't catch it. Should not the indentation be 4? lineWrappingIndentation is not set in checkstyle.xml, I found default is 4 in https://checkstyle.sourceforge.io/config_misc.html.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214305205
@zhengchenyu Thanks a lot for your contribution, need to fix some checkstyle issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945259806
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import static org.junit.Assert.fail;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
Review Comment:
Of couse could. But is it necessary? Here just construct a easy object!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945404189
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##########
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
- this.rmAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
- this.ahsAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException, IOException {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This code should show like this. If refactor is only for clean code. I think it is not very clean. Maybe I did not catch your idea clearly.
```
ServletContext context = getServletContext();
AppReportFetcher report =
((AppReportFetcher) context.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r950986663
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+ return StringHelper.pjoin(scheme + subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws YarnException, IOException {
+ if (subClusters.containsKey(scId)) {
+ return;
+ }
+ SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+ Configuration subClusterConf = new Configuration(getConf());
+ FederationProxyProviderUtil
+ .updateConfForFederation(subClusterConf, subClusterInfo.getSubClusterId().toString());
Review Comment:
Of course not. but explicit call is not wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1161337966
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 59s | | trunk passed |
| +1 :green_heart: | compile | 4m 45s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 58s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 36s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 2s | | the patch passed |
| +1 :green_heart: | compile | 4m 27s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 27s | | the patch passed |
| +1 :green_heart: | compile | 3m 33s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 15s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 11 new + 106 unchanged - 3 fixed = 117 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 55s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 56s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 6s | | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 1m 34s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/9/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch passed. |
| -1 :x: | unit | 106m 30s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/9/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 254m 28s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2501882c45b8 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e1bec4858141853e174390546ed520fef245331b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/9/testReport/ |
| Max. process+thread count | 872 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1185262925
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 33s | | trunk passed |
| +1 :green_heart: | compile | 4m 31s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 44s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 13s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 9s | | the patch passed |
| +1 :green_heart: | compile | 4m 12s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 12s | | the patch passed |
| +1 :green_heart: | compile | 3m 32s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 22s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/14/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 10 new + 106 unchanged - 3 fixed = 116 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 5s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 8s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-server-web-proxy in the patch passed. |
| -1 :x: | unit | 108m 13s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/14/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 249m 55s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b04715004e54 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 008598bd31132dd31b4dee87a5a487dd3f6a3ed7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/14/testReport/ |
| Max. process+thread count | 1000 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1675571426
@goiri Thank you very much for your help in reviewing the code! @zhengchenyu Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1184199779
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 27s | | trunk passed |
| +1 :green_heart: | compile | 4m 20s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 5s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 9s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 14s | | the patch passed |
| +1 :green_heart: | compile | 4m 32s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 32s | | the patch passed |
| +1 :green_heart: | compile | 3m 49s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 16s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/13/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 106 unchanged - 3 fixed = 107 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 47s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 54s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 21s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 101m 49s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 251m 46s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 657a61b140b3 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 39cfa39c6ef9008b852ae76aaacf48f74c4d9396 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/13/testReport/ |
| Max. process+thread count | 936 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r903230413
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServerFed.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+
+public class TestWebAppProxyServerFed extends TestWebAppProxyServer {
+
+ @Override
+ public void setUp() throws Exception {
+ super.setUp();
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
Review Comment:
@goiri I remove TestWebProxyServerFed. It is unnecessary for us to test startup. I add new UT TestWebAppProxyServletFed, will test proxy link in federation mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
URL: https://github.com/apache/hadoop/pull/4314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r889778464
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServerFed.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+
+public class TestWebAppProxyServerFed extends TestWebAppProxyServer {
+
+ @Override
+ public void setUp() throws Exception {
+ super.setUp();
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
Review Comment:
It's only for test the basic function (For example: start or stop and so on) of WebAppProxyServer in federation mode.
For me, I think TestWebAppProxyServer is not generic enough, because that UT only test start, stop and bindaddress. I don't know whether is necessary to add some integration testing so that we can test proxy app web link?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu closed pull request #4314: YARN-11153. Make proxy server support yarn federation.
URL: https://github.com/apache/hadoop/pull/4314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956166342
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##########
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
- this.rmAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
- this.ahsAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException, IOException {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
For example, it can be extracted here to make the code easier to read. I think the first way you provide is feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956841060
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##########
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
- this.rmAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
- this.ahsAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException, IOException {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
Thanks for review, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r944610371
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##########
@@ -361,8 +362,8 @@ public void testAppReportForEmptyTrackingUrl() throws Exception {
proxyConn.getURL().toString().equals(appAddressInRm));
//set AHS_ENBALED = true to simulate getting the app report from AHS
- configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
- true);
+ configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ proxy.proxy.appReportFetcher.setAhsAppPageUrlBase(configuration);
Review Comment:
Not sure why checkstyle complains about this.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##########
@@ -375,8 +376,8 @@ public void testAppReportForEmptyTrackingUrl() throws Exception {
String appAddressInAhs = WebAppUtils.getHttpSchemePrefix(configuration) +
WebAppUtils.getAHSWebAppURLWithoutScheme(configuration) +
"/applicationhistory" + "/app/" + app.toString();
- assertTrue("Webapp proxy servlet should have redirected to AHS",
- proxyConn.getURL().toString().equals(appAddressInAhs));
+ assertEquals("Webapp proxy servlet should have redirected to AHS", appAddressInAhs,
Review Comment:
I think the indentation is broken here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945245127
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import static org.junit.Assert.fail;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade = FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1, subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2, subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1.getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2.getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws Exception {
+ testHelper(false);
+
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ LambdaTestUtils.intercept(ApplicationNotFoundException.class, appNotFoundExceptionMsg,
+ () -> fetcher.getApplicationReport(appId1));
+ LambdaTestUtils.intercept(ApplicationNotFoundException.class, appNotFoundExceptionMsg,
+ () -> fetcher.getApplicationReport(appId2));
+
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Assert.assertNull("HistoryManager should be null as AHS is disabled", history);
+ }
+
+ @Test
+ public void testGetRmAppPageUrlBase() throws IOException, YarnException {
+ testHelper(true);
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ Assert.assertEquals(fetcher.getRmAppPageUrlBase(appId1),
+ StringHelper.pjoin(scheme + clusterInfo1.getRMWebServiceAddress(), "cluster", "app"));
+ Assert.assertEquals(fetcher.getRmAppPageUrlBase(appId2),
+ StringHelper.pjoin(scheme + clusterInfo2.getRMWebServiceAddress(), "cluster", "app"));
+ }
+
+ static class FedAppReportFetcherForTest extends FedAppReportFetcher {
+
+ FedAppReportFetcherForTest(Configuration conf) {
+ super(conf);
+ }
+
+ @Override
+ protected ApplicationHistoryProtocol getAHSProxy(Configuration conf)
+ throws IOException {
+ GetApplicationReportResponse resp = Mockito.mock(GetApplicationReportResponse.class);
+ history = Mockito.mock(ApplicationHistoryProtocol.class);
+ try {
+ Mockito.when(history.getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenReturn(resp);
+ } catch (YarnException e) {
+ // This should never happen
+ fail("Found exception when getApplicationReport!");
+ }
+ return history;
+ }
+ }
+
Review Comment:
remove blank lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1674237206
@goiri Can you double check this pr again? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1161670272
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 38s | | trunk passed |
| +1 :green_heart: | compile | 4m 23s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 48s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | | the patch passed |
| +1 :green_heart: | compile | 3m 50s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 50s | | the patch passed |
| +1 :green_heart: | compile | 3m 23s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 8s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/10/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2 new + 106 unchanged - 3 fixed = 108 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 1s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 58s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 0s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 101m 30s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 235m 38s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c55cb47c2581 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34049e7c3e925e4574719e2c6592dd340e09d39a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/10/testReport/ |
| Max. process+thread count | 944 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1128578603
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 21s | | trunk passed |
| +1 :green_heart: | compile | 4m 26s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 55s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 4m 15s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 15s | | the patch passed |
| +1 :green_heart: | compile | 3m 38s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 20s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 11 new + 95 unchanged - 4 fixed = 106 total (was 99) |
| +1 :green_heart: | mvnsite | 1m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 107m 11s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 239m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 761a1c6f654d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6f15363c6511df156cc7fdab7ba4ae7895043420 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/2/testReport/ |
| Max. process+thread count | 934 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945302648
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##########
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
- this.rmAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
- this.ahsAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException, IOException {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This line of code needs to be refactored, too many layers and not easy to read.
This way of writing might be better.
```
AppReportFetcher appReportFetcher = (AppReportFetcher) getServletContext();
Attribute attr1 = appReportFetcher.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE);
return attr1.getAhsAppPageUrlBase();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1188373132
Anybody else available for a review?
Otherwise we go with this by the end of the week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1186984996
@goiri How about merge trunk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r928791677
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java:
##########
@@ -0,0 +1,366 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.HttpURLConnection;
+import java.net.URI;
+import java.net.URL;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer2;
+import org.apache.hadoop.security.authorize.AccessControlList;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.eclipse.jetty.server.Server;
+import org.eclipse.jetty.server.ServerConnector;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test the WebAppProxyServlet and WebAppProxy. For back end use simple web server.
+ */
+public class TestWebAppProxyServletFed {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TestWebAppProxyServletFed.class);
+
+ public static final String AM_PREFIX = "AM";
+ public static final String RM_PREFIX = "RM";
+ public static final String AHS_PREFIX = "AHS";
+
+ /*
+ * Mocked Server is used for simulating the web of AppMaster, ResourceMamanger or TimelineServer.
+ * */
+ private static Server mockServer;
+ private static int mockServerPort = 0;
+
+ /**
+ * Simple http server. Server should send answer with status 200
+ */
+ @BeforeClass
+ public static void setUp() throws Exception {
+ mockServer = new Server(0);
+ ((QueuedThreadPool) mockServer.getThreadPool()).setMaxThreads(20);
+ ServletContextHandler context = new ServletContextHandler();
+ context.setContextPath("/");
+ context.addServlet(new ServletHolder(new MockWebServlet(AM_PREFIX)), "/amweb/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(RM_PREFIX)), "/cluster/app/*");
+ context.addServlet(new ServletHolder(new MockWebServlet(AHS_PREFIX)),
+ "/applicationhistory/app/*");
+ mockServer.setHandler(context);
+
+ ((ServerConnector) mockServer.getConnectors()[0]).setHost("localhost");
+ mockServer.start();
+ mockServerPort = ((ServerConnector) mockServer.getConnectors()[0]).getLocalPort();
+ LOG.info("Running embedded servlet container at: http://localhost:" + mockServerPort);
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ if (mockServer != null) {
+ mockServer.stop();
+ mockServer.destroy();
+ mockServer = null;
+ }
+ }
+
+ @Test
+ public void testWebServlet() throws IOException {
+ HttpURLConnection conn;
+ // 1. Mocked AppMaster web Test
+ URL url = new URL("http", "localhost", mockServerPort, "/amweb/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 2. Mocked RM web Test
+ url = new URL("http", "localhost", mockServerPort, "/cluster/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(RM_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+
+ // 3. Mocked AHS web Test
+ url = new URL("http", "localhost", mockServerPort, "/applicationhistory/app/apptest");
+ conn = (HttpURLConnection) url.openConnection();
+ conn.connect();
+ assertEquals(HttpURLConnection.HTTP_OK, conn.getResponseCode());
+ assertEquals(AHS_PREFIX + "/apptest", readResponse(conn));
+ conn.disconnect();
+ }
+
+ @Test(timeout=5000)
+ public void testWebAppProxyServletFed() throws Exception {
+
+ Configuration conf = new Configuration();
+ conf.set(YarnConfiguration.PROXY_ADDRESS, "localhost:9090");
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, true);
+ conf.set(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS, "localhost:" + mockServerPort);
+ // overriding num of web server threads, see HttpServer.HTTP_MAXTHREADS
+ conf.setInt("hadoop.http.max.threads", 10);
+
+ // Create sub cluster information.
+ SubClusterId subClusterId1 = SubClusterId.newInstance("scid1");
+ SubClusterId subClusterId2 = SubClusterId.newInstance("scid2");
+ SubClusterInfo subClusterInfo1 = SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1",
+ "10.0.0.1:1", "10.0.0.1:1", "localhost:" + mockServerPort, SubClusterState.SC_RUNNING, 0,
+ "");
+ SubClusterInfo subClusterInfo2 = SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1",
+ "10.0.0.2:1", "10.0.0.2:1", "10.0.0.2:1", SubClusterState.SC_RUNNING, 0, "");
+
+ // App1 and App2 is running applications.
+ ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+ String appUrl1 = "http://localhost:" + mockServerPort + "/amweb/" + appId1;
+ String appUrl2 = "http://localhost:" + mockServerPort + "/amweb/" + appId2;
+ // App3 is accepted application, has not registered original url to am.
+ ApplicationId appId3 = ApplicationId.newInstance(0, 3);
+ // App4 is finished application, has remove from rm, but not remove from timeline server.
+ ApplicationId appId4 = ApplicationId.newInstance(0, 4);
+
+ // Mock for application
+ ApplicationClientProtocol appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId1)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId1, YarnApplicationState.RUNNING, appUrl1)));
+ Mockito.when(appManager1.getApplicationReport(GetApplicationReportRequest.newInstance(appId3)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId3, YarnApplicationState.ACCEPTED, null)));
+
+ ApplicationClientProtocol appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2.getApplicationReport(GetApplicationReportRequest.newInstance(appId2)))
+ .thenReturn(GetApplicationReportResponse
+ .newInstance(newApplicationReport(appId2, YarnApplicationState.RUNNING, appUrl2)));
Review Comment:
indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r928787228
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##########
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
Review Comment:
Should we add the java doc？Why remove the comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669545487
@zhengchenyu Thanks for the contribution! LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945404189
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##########
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
- this.rmAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
- this.ahsAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException, IOException {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This code should show like this. If refactor is only for clean code. I think it is not very clean. Maybe I did not catch your idea clearly.
```
ServletContext context = getServletContext();
AppReportFetcher report =
((AppReportFetcher) context.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
or
```
AppReportFetcher report =
((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214700469
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 17s | | trunk passed |
| +1 :green_heart: | compile | 4m 7s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 12s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 3m 47s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 13s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 74 unchanged - 24 fixed = 74 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 34s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 102m 45s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not generate ASF License warnings. |
| | | 226m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b7888fa7415e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 75d7994581c9bf7e62b468b0c8f21fe86460ecf2 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/23/testReport/ |
| Max. process+thread count | 982 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/23/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1184335341
@goiri Sorry, I use wrong check style, I have fix it. Can you please review this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r889435983
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServerFed.java:
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+
+public class TestWebAppProxyServerFed extends TestWebAppProxyServer {
+
+ @Override
+ public void setUp() throws Exception {
+ super.setUp();
+ conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
Review Comment:
Do we want to extend some tests just for federation or TestWebAppProxyServer is now generic enough?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ Configuration conf;
+ static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ try {
+ fetcher.getApplicationReport(appId2);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ if (history != null) {
Review Comment:
assertNotNull and have the message there
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##########
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestFedAppReportFetcher {
+
+ Configuration conf;
+ static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 = SubClusterInfo
+ .newInstance(subClusterId1, "10.0.0.1:1000", "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private SubClusterInfo clusterInfo2 = SubClusterInfo
+ .newInstance(subClusterId2, "10.0.0.2:1000", "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", SubClusterState.SC_RUNNING, 0l, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+ history = null;
+ fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+ conf = new YarnConfiguration();
+ conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+ isAHSEnabled);
+
+ FederationStateStoreFacade fedFacade =
+ FederationStateStoreFacade.getInstance();
+ FederationStateStore fss = new MemoryFederationStateStore();
+ fss.init(conf);
+ fedFacade.reinitialize(fss, conf);
+
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+ fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId1,
+ subClusterId1)));
+ fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+ .newInstance(ApplicationHomeSubCluster.newInstance(appId2,
+ subClusterId2)));
+
+ appManager1 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager1
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ appManager2 = Mockito.mock(ApplicationClientProtocol.class);
+ Mockito.when(appManager2
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class)))
+ .thenThrow(new ApplicationNotFoundException(appNotFoundExceptionMsg));
+
+ fetcher = new TestFedAppReportFetcher.FedAppReportFetcherForTest(conf);
+ fetcher.registerSubCluster(clusterInfo1, appManager1);
+ fetcher.registerSubCluster(clusterInfo2, appManager2);
+ }
+
+ @Test
+ public void testFetchReportAHSEnabled() throws YarnException, IOException {
+ testHelper(true);
+ fetcher.getApplicationReport(appId1);
+ fetcher.getApplicationReport(appId2);
+ Mockito.verify(history, Mockito.times(2))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager1, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ Mockito.verify(appManager2, Mockito.times(1))
+ .getApplicationReport(Mockito.any(GetApplicationReportRequest.class));
+ }
+
+ @Test
+ public void testFetchReportAHSDisabled() throws YarnException, IOException {
+ testHelper(false);
+ try {
+ fetcher.getApplicationReport(appId1);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
+ /* RM will not know of the app and Application History Service is disabled
+ * So we will not try to get the report from AHS and RM will throw
+ * ApplicationNotFoundException
+ */
+ }
+ try {
+ fetcher.getApplicationReport(appId2);
+ } catch (ApplicationNotFoundException e) {
+ Assert.assertTrue(e.getMessage() == appNotFoundExceptionMsg);
Review Comment:
== will give issues right? assertEquals()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1177006408
@goiri Is this proposal OK? YARN-11154 is blocked by this. Can you please review this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1193902708
@goiri @zhengchenyu this pr looks good, some checkstyles need fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support yarn federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1160323672
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 28s | | trunk passed |
| +1 :green_heart: | compile | 3m 59s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 40s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 28s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvninstall | 0m 38s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 1m 55s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 1m 55s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 47s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 47s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 12s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 10 new + 107 unchanged - 2 fixed = 117 total (was 109) |
| -1 :x: | mvnsite | 0m 31s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvnsite | 0m 40s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | javadoc | 0m 36s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javadoc | 0m 46s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 343 unchanged - 0 fixed = 344 total (was 343) |
| -1 :x: | spotbugs | 0m 29s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | spotbugs | 0m 41s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | shadedclient | 8m 57s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 59s | | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 0m 35s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | unit | 0m 45s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. |
| | | 115m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5a8328502faa 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 852b7f1dff16cf60d43711bc70d33ad60c30aa76 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/testReport/ |
| Max. process+thread count | 558 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r950851572
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map<SubClusterId, Pair<SubClusterInfo, ApplicationClientProtocol>> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+ super(conf);
+ subClusters = new ConcurrentHashMap<>();
+ federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+ ApplicationClientProtocol applicationsManager = subClusters.get(scid).getRight();
+
+ return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+ SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+ createSubclusterIfAbsent(scid);
+
+ SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+ String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+ return StringHelper.pjoin(scheme + subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws YarnException, IOException {
+ if (subClusters.containsKey(scId)) {
+ return;
+ }
+ SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+ Configuration subClusterConf = new Configuration(getConf());
+ FederationProxyProviderUtil
+ .updateConfForFederation(subClusterConf, subClusterInfo.getSubClusterId().toString());
Review Comment:
subClusterInfo.getSubClusterId() should be enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956166577
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##########
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
- this.rmAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
- this.ahsAppPageUrlBase =
- StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException, IOException {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
+ .getRmAppPageUrlBase(id);
+ }
+
+ private String getAhsAppPageUrlBase() {
+ return ((AppReportFetcher) getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
extracted too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1228639768
Thank you very much for your contribution, overall it looks ok, but some code is better to extract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1229823333
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 31s | | trunk passed |
| +1 :green_heart: | compile | 4m 3s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 25s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 48s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | compile | 3m 15s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 12s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 74 unchanged - 24 fixed = 74 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 11s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 117m 39s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. |
| | | 239m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/24/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b6d57850ef99 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ada446b9d7d9ddc81e4521bf9c9f9a28ad241dbe |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/24/testReport/ |
| Max. process+thread count | 923 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/24/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214265428
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 3m 46s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 14s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 3s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 56s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 17s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvninstall | 0m 31s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 1m 44s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 1m 44s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 36s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 36s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 30s | [/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | The patch fails to run checkstyle in hadoop-yarn-server |
| -1 :x: | mvnsite | 0m 17s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvnsite | 0m 31s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | javadoc | 0m 16s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server-web-proxy in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javadoc | 0m 28s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javadoc | 0m 17s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-server-web-proxy in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javadoc | 0m 39s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 343 unchanged - 0 fixed = 344 total (was 343) |
| -1 :x: | spotbugs | 0m 18s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | spotbugs | 0m 32s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | shadedclient | 8m 47s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 51s | | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 0m 17s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | unit | 0m 33s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 105m 58s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5de0f93b038c 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15241379806c86275c7f14f8602ede7c74b76aae |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/testReport/ |
| Max. process+thread count | 682 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214164427
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 25s | | trunk passed |
| +1 :green_heart: | compile | 4m 4s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 37s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 58s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 25s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 3m 47s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 21s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 15s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 85 unchanged - 24 fixed = 85 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 1s | | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 56s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 4s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 1m 29s | | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 102m 29s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 238m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3425f047a000 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 05854d89656769840416945c719509ac13ceeddd |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/19/testReport/ |
| Max. process+thread count | 952 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669019094
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 50s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 7s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 1m 34s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 1m 1s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/26/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 97m 30s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not generate ASF License warnings. |
| | | 205m 32s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/26/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6a9cc211415e 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0c3870d92e574efee462dae935296702642ddcaa |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/26/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/26/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1668903090
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 34s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 50s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 13s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvninstall | 0m 20s | [/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 1m 2s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt) | hadoop-yarn-server in the patch failed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 1m 2s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt) | hadoop-yarn-server in the patch failed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 0m 54s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | javac | 0m 54s | [/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt) | hadoop-yarn-server in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 21s | [/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt) | The patch fails to run checkstyle in hadoop-yarn-server |
| -1 :x: | mvnsite | 0m 15s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvnsite | 0m 22s | [/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | javadoc | 0m 23s | [/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt) | hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 29s | [/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt) | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05 with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) |
| -1 :x: | spotbugs | 0m 14s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | spotbugs | 0m 22s | [/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | shadedclient | 9m 56s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 17s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt) | hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | unit | 0m 28s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not generate ASF License warnings. |
| | | 86m 8s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9a56780fe1a9 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / db51c7a90c72480930f791b96649564d3c69a9cd |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/testReport/ |
| Max. process+thread count | 554 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri merged pull request #4314: YARN-11153. Make proxy server support YARN federation.
Posted by "goiri (via GitHub)" <gi...@apache.org>.
goiri merged PR #4314:
URL: https://github.com/apache/hadoop/pull/4314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org