You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/09/02 11:32:44 UTC
[GitHub] [hadoop] slfan1989 opened a new pull request, #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
slfan1989 opened a new pull request, #4846:
URL: https://github.com/apache/hadoop/pull/4846
JIRA: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
1. Currently this interface returns the number of apps in all sub-clusters, increasing the limit on the number of query apps, limited to 1000 apps.
2. Allows to query the App based on the specified HomeSubCluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r972467705
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,24 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM(SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(order by [createTime] desc) AS app_rank
+ FROM [dbo].[applicationsHomeSubCluster]
+ WHERE [homeSubCluster] = @homeSubCluster OR @homeSubCluster = '') AS t
Review Comment:
In the other one we use AS applicationshomesubcluster; let's try to be consistent with the names as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r970127749
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,21 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ *,
+ @rownum := 0,
+ IF(homeSubCluster_IN = '', 1, (homeSubCluster = homeSubCluster_IN)) AS filter_result
+ FROM applicationshomesubcluster
+ ORDER BY createTime DESC) AS app_home_sc
+ WHERE filter_result = 1
Review Comment:
Why you cannot do:
```
SELECT
applicationId,
homeSubCluster,
createTime
FROM applicationshomesubcluster
WHERE homeSubCluster_IN = '' OR homeSubCluster = homeSubCluster_IN
```
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,37 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ IF @homeSubCluster = ''
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM(SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(order by [createTime] desc) AS app_rank
+ FROM [dbo].[applicationsHomeSubCluster]) AS t
+ WHERE app_rank <= @limit;
+ ELSE
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM(SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(partition by [homeSubCluster] order by [createTime] desc) AS app_rank
+ FROM [dbo].[applicationsHomeSubCluster]
+ WHERE [homeSubCluster] = @homeSubCluster) AS t
Review Comment:
Why not
```
WHERE [homeSubCluster] = @homeSubCluster OR @homeSubCluster = ''
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r968842977
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,26 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ applicationId,
+ homeSubCluster,
+ createTime,
+ @app_rank := IF(@home_sc = homeSubCluster, @app_rank + 1, 1) AS app_rank,
+ @home_sc := homeSubCluster
+ FROM applicationshomesubcluster
Review Comment:
The tabs in line warnings are legit. Let's replace them with spaces.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/ApplicationHomeSubCluster.java:
##########
@@ -51,6 +51,18 @@ public static ApplicationHomeSubCluster newInstance(ApplicationId appId,
return appMapping;
}
+ @Private
+ @Unstable
+ public static ApplicationHomeSubCluster newInstance(ApplicationId appId, long createTime,
+ SubClusterId homeSubCluster) {
+ ApplicationHomeSubCluster appMapping =
Review Comment:
Single line
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/FederationStateStoreBaseTest.java:
##########
@@ -410,6 +414,89 @@ public void testGetApplicationsHomeSubCluster() throws Exception {
Assert.assertTrue(result.getAppsHomeSubClusters().contains(ahsc2));
}
+ @Test
+ public void testGetApplicationsHomeSubClusterEmpty() throws Exception {
+ LambdaTestUtils.intercept(YarnException.class,
+ "Missing getApplicationsHomeSubCluster request",
+ () -> stateStore.getApplicationsHomeSubCluster(null));
+ }
+
+ @Test
+ public void testGetApplicationsHomeSubClusterFilter() throws Exception {
+ // Add ApplicationHomeSC - SC1
+ long now = Time.now();
+
+ Set<ApplicationHomeSubCluster> appHomeSubClusters = new HashSet<>();
+
+ for (int i = 0; i < TEN_ROUNDS; i++) {
Review Comment:
TEN_ROUNDS is a weird variable.
I was initially thinking more about something like NUM_APPS_X and NUM_APSS_Y
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,26 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ applicationId,
+ homeSubCluster,
+ createTime,
+ @app_rank := IF(@home_sc = homeSubCluster, @app_rank + 1, 1) AS app_rank,
+ @home_sc := homeSubCluster
+ FROM applicationshomesubcluster
+ ORDER BY createTime DESC
+ ) ranked
+ WHERE app_rank <= limit_IN
+ AND (CASE WHEN t.homeSubCluster_IN IS NULL THEN 1 = 1
Review Comment:
Can we make this a regular boolean condition with no CASE?
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,26 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ applicationId,
+ homeSubCluster,
+ createTime,
Review Comment:
Shouldn't we sort this?
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,27 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM
+ (SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(partition by [homeSubCluster] order by [createTime] desc) AS row_num
+ FROM [dbo].[applicationsHomeSubCluster]) AS t
+ WHERE row_num <= @limit
+ AND (CASE WHEN @homeSubCluster IS NULL THEN 1
Review Comment:
Do we really need to use case? We cannot do some boolean condition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1245417267
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 35s | | trunk passed |
| +1 :green_heart: | compile | 9m 55s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 46s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 46s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 18s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 12s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 35s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 56s | | the patch passed |
| +1 :green_heart: | compile | 9m 4s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 4s | | the patch passed |
| +1 :green_heart: | javac | 9m 4s | | the patch passed |
| +1 :green_heart: | compile | 8m 36s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 36s | | the patch passed |
| +1 :green_heart: | javac | 8m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 9s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 36s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 55s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 215m 36s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/20/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch failed. |
| +1 :green_heart: | unit | 1m 29s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 33s | | hadoop-yarn-common in the patch passed. |
| -1 :x: | unit | 3m 28s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/20/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 35s | | The patch does not generate ASF License warnings. |
| | | 435m 14s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 7f0f3ddc0236 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 411d4d2af5b564ecaa90c3688ccdfcf7a6d2047a |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/20/testReport/ |
| Max. process+thread count | 2626 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/20/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964250594
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,24 +261,52 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- long start = clock.getTime();
- List<ApplicationHomeSubCluster> result = new ArrayList<>();
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
try {
- for (String child : zkManager.getChildren(appsZNode)) {
- ApplicationId appId = ApplicationId.fromString(child);
- SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
- result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List<String> children = zkManager.getChildren(appsZNode);
+ List<ApplicationHomeSubCluster> result =
+ children.stream().map(child -> generateAppHomeSC(child))
+ .filter(appHomeSC -> judgeAdd(requestSC, appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+ long end = clock.getTime();
+ opDurations.addGetAppsHomeSubClusterDuration(start, end);
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC, result.size());
+ return GetApplicationsHomeSubClusterResponse.newInstance(result);
} catch (Exception e) {
String errMsg = "Cannot get apps: " + e.getMessage();
FederationStateStoreUtils.logAndThrowStoreException(LOG, errMsg);
}
- long end = clock.getTime();
- opDurations.addGetAppsHomeSubClusterDuration(start, end);
- return GetApplicationsHomeSubClusterResponse.newInstance(result);
+
+ throw new YarnException("Cannot get app by request");
+ }
+
+ private ApplicationHomeSubCluster generateAppHomeSC(String appId) {
+ try {
+ ApplicationId applicationId = ApplicationId.fromString(appId);
+ SubClusterId homeSubCluster = getApp(applicationId);
+ ApplicationHomeSubCluster app =
+ ApplicationHomeSubCluster.newInstance(applicationId, homeSubCluster);
+ return app;
+ } catch (Exception ex) {
+ LOG.error("get homeSubCluster by appId = {}.", appId);
+ }
+ return null;
+ }
+
+ private boolean judgeAdd(SubClusterId filterSubCluster, SubClusterId homeSubCluster) {
Review Comment:
I will re-select an appropriate method name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r970008365
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,26 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ applicationId,
+ homeSubCluster,
+ createTime,
Review Comment:
We can sort internally and get the sorted result directly externally.
```
CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
BEGIN
SELECT
applicationId,
homeSubCluster,
createTime
-- In this part, we can get the results directly, which are already sorted
FROM
(SELECT
*,
@rownum := 0,
IF(homeSubCluster_IN = '', 1, (homeSubCluster = homeSubCluster_IN)) AS filter_result
FROM applicationshomesubcluster
-- In this part we have sorted
ORDER BY createTime DESC) AS app_home_sc
WHERE filter_result = 1
AND (@rownum := @rownum + 1) <= limit_IN;
END //
```
Case1: parameters `limit_IN = 10 homeSubCluster_IN = ''`, We expect to get 10 records in reverse order.
<img width="724" alt="image" src="https://user-images.githubusercontent.com/55643692/189991492-dbb3d63f-578a-4b58-ad06-0bb52074cd0c.png">
Case2: parameters `limit_IN = 10 homeSubCluster_IN = 'SC-1'`, We expect to get 5 records in reverse order.
<img width="629" alt="image" src="https://user-images.githubusercontent.com/55643692/189992078-5b056737-8e91-48a1-bce1-597f757877ab.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1236867158
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 42s | | trunk passed |
| +1 :green_heart: | compile | 10m 16s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 13s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 46s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 56s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 40s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 39s | | the patch passed |
| +1 :green_heart: | compile | 9m 31s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 31s | | the patch passed |
| +1 :green_heart: | javac | 9m 31s | | the patch passed |
| +1 :green_heart: | compile | 9m 1s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 1s | | the patch passed |
| +1 :green_heart: | javac | 9m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 54s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 165 unchanged - 0 fixed = 168 total (was 165) |
| +1 :green_heart: | mvnsite | 8m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 48s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 10s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 237m 14s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 35s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 21s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 28s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 13s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 37s | | The patch does not generate ASF License warnings. |
| | | 471m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 1eedc12c0190 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eecc58871e86f20f513c7499777785f628f12367 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/5/testReport/ |
| Max. process+thread count | 2775 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1243947248
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 38s | | trunk passed |
| +1 :green_heart: | compile | 10m 33s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 14s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 13s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 16s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 53s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 16s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 28s | | the patch passed |
| +1 :green_heart: | compile | 10m 5s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 10m 5s | | the patch passed |
| +1 :green_heart: | javac | 10m 5s | | the patch passed |
| +1 :green_heart: | compile | 9m 18s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 18s | | the patch passed |
| +1 :green_heart: | javac | 9m 18s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/19/artifact/out/blanks-tabs.txt) | The patch 3 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 1m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 42s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 50s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 0s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 212m 43s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/19/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch failed. |
| +1 :green_heart: | unit | 1m 30s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 31s | | hadoop-yarn-common in the patch passed. |
| -1 :x: | unit | 3m 29s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/19/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 17s | | The patch does not generate ASF License warnings. |
| | | 439m 21s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 5ce6cecabac6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1d000209b4845a7bae18ddcda5ca5ff147454fc2 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/19/testReport/ |
| Max. process+thread count | 2478 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1250132393
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 38s | | trunk passed |
| +1 :green_heart: | compile | 9m 51s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 48s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 48s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 14s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 45s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 13s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 59s | | the patch passed |
| +1 :green_heart: | compile | 11m 0s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 11m 0s | | the patch passed |
| +1 :green_heart: | javac | 11m 0s | | the patch passed |
| +1 :green_heart: | compile | 9m 19s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 19s | | the patch passed |
| +1 :green_heart: | javac | 9m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 18s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 48s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 34s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 236m 35s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 31s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 28s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not generate ASF License warnings. |
| | | 455m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/25/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 432c09e8bf8e 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bd195f2eb021ed271cff0cd46bafe339d07e4a2e |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/25/testReport/ |
| Max. process+thread count | 2773 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/25/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r962057944
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,23 +260,41 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
long start = clock.getTime();
List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
try {
- for (String child : zkManager.getChildren(appsZNode)) {
+ List<String> childrens = zkManager.getChildren(appsZNode);
+ for (String child : childrens) {
+ if (appCount >= maxAppsInStateStore) {
+ break;
+ }
ApplicationId appId = ApplicationId.fromString(child);
SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
+ // If the requestSubClusterId that needs to be filtered in the request
+ // is inconsistent with the SubClusterId in the data, continue to the next round
+ if (requestSubClusterId != null && !requestSubClusterId.equals(homeSubCluster)) {
+ continue;
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1239121352
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 30s | | trunk passed |
| +1 :green_heart: | compile | 10m 30s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 57s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 50s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 55s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 20m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 14s | | the patch passed |
| +1 :green_heart: | compile | 9m 30s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 30s | | the patch passed |
| +1 :green_heart: | javac | 9m 30s | | the patch passed |
| +1 :green_heart: | compile | 9m 27s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 27s | | the patch passed |
| +1 :green_heart: | javac | 9m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 8s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 14s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 20m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 19s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 239m 15s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 39s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 26s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 31s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 18s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 23s | | The patch does not generate ASF License warnings. |
| | | 479m 11s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux a2a64fc272a9 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7de66bf8691474238aa262d25da08ee26a4fe028 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/11/testReport/ |
| Max. process+thread count | 2772 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964257610
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java:
##########
@@ -90,6 +95,7 @@ protected ApplicationClientProtocol getClientRMProxyForSubCluster(
mockRMs.put(subClusterId, mockRM);
}
initNodeAttributes(subClusterId, mockRM);
+ initReservationSystem(mockRM);
Review Comment:
The new pr will optimize the repetitive submission of reservations and refactor some test code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964256887
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java:
##########
@@ -90,6 +95,7 @@ protected ApplicationClientProtocol getClientRMProxyForSubCluster(
mockRMs.put(subClusterId, mockRM);
}
initNodeAttributes(subClusterId, mockRM);
+ initReservationSystem(mockRM);
Review Comment:
I will submit pr separately to improve this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1259987263
@goiri Can you help merge this pr into trunk branch? Thank you very much! after this pr is completed, I will submit a new pr to clean up the completed application in the router's StateStore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1247192785
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 39s | | trunk passed |
| +1 :green_heart: | compile | 10m 43s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 1s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 0s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 7s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 57s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 21s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 23s | | the patch passed |
| +1 :green_heart: | compile | 9m 52s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 52s | | the patch passed |
| +1 :green_heart: | javac | 9m 52s | | the patch passed |
| +1 :green_heart: | compile | 9m 17s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 17s | | the patch passed |
| +1 :green_heart: | javac | 9m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 35s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 58s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 241m 32s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/23/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 35s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 35s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 37s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 25s | | The patch does not generate ASF License warnings. |
| | | 471m 2s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 4158d7edc970 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 490cf3c017704b38b5a193df7159d09164096507 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/23/testReport/ |
| Max. process+thread count | 2773 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/23/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r970137060
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,21 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ *,
+ @rownum := 0,
+ IF(homeSubCluster_IN = '', 1, (homeSubCluster = homeSubCluster_IN)) AS filter_result
+ FROM applicationshomesubcluster
+ ORDER BY createTime DESC) AS app_home_sc
+ WHERE filter_result = 1
Review Comment:
Thank you very much for your suggestion, I agree with you, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964122414
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,24 +261,52 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- long start = clock.getTime();
- List<ApplicationHomeSubCluster> result = new ArrayList<>();
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
try {
- for (String child : zkManager.getChildren(appsZNode)) {
- ApplicationId appId = ApplicationId.fromString(child);
- SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
- result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List<String> children = zkManager.getChildren(appsZNode);
+ List<ApplicationHomeSubCluster> result =
+ children.stream().map(child -> generateAppHomeSC(child))
+ .filter(appHomeSC -> judgeAdd(requestSC, appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+ long end = clock.getTime();
+ opDurations.addGetAppsHomeSubClusterDuration(start, end);
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC, result.size());
+ return GetApplicationsHomeSubClusterResponse.newInstance(result);
} catch (Exception e) {
String errMsg = "Cannot get apps: " + e.getMessage();
FederationStateStoreUtils.logAndThrowStoreException(LOG, errMsg);
}
- long end = clock.getTime();
- opDurations.addGetAppsHomeSubClusterDuration(start, end);
- return GetApplicationsHomeSubClusterResponse.newInstance(result);
+
+ throw new YarnException("Cannot get app by request");
+ }
+
+ private ApplicationHomeSubCluster generateAppHomeSC(String appId) {
+ try {
+ ApplicationId applicationId = ApplicationId.fromString(appId);
+ SubClusterId homeSubCluster = getApp(applicationId);
+ ApplicationHomeSubCluster app =
+ ApplicationHomeSubCluster.newInstance(applicationId, homeSubCluster);
+ return app;
+ } catch (Exception ex) {
+ LOG.error("get homeSubCluster by appId = {}.", appId);
+ }
+ return null;
+ }
+
+ private boolean judgeAdd(SubClusterId filterSubCluster, SubClusterId homeSubCluster) {
Review Comment:
`judgeAdd` is a weird name fot this function.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/FederationStateStoreBaseTest.java:
##########
@@ -410,6 +412,89 @@ public void testGetApplicationsHomeSubCluster() throws Exception {
Assert.assertTrue(result.getAppsHomeSubClusters().contains(ahsc2));
}
+ @Test
+ public void testGetApplicationsHomeSubClusterEmpty() throws Exception {
+ LambdaTestUtils.intercept(YarnException.class,
+ "Missing getApplicationsHomeSubCluster request",
+ () -> stateStore.getApplicationsHomeSubCluster(null));
+ }
+
+ @Test
+ public void testGetApplicationsHomeSubClusterFilter() throws Exception {
+ // Add ApplicationHomeSC - SC1
+ long now = Time.now();
+
+ Set<ApplicationHomeSubCluster> appHomeSubClusters = new HashSet<>();
+
+ for (int i = 0; i < 10; i++) {
Review Comment:
Make 10 and 20 constants.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java:
##########
@@ -90,6 +95,7 @@ protected ApplicationClientProtocol getClientRMProxyForSubCluster(
mockRMs.put(subClusterId, mockRM);
}
initNodeAttributes(subClusterId, mockRM);
+ initReservationSystem(mockRM);
Review Comment:
This seems out of scope.
Should we do it in a separate PR?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,17 +261,36 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
}
- GetApplicationsHomeSubClusterResponse.newInstance(result);
+ SubClusterId requestSC = request.getSubClusterId();
+ List<ApplicationHomeSubCluster> result = applications.keySet().stream()
+ .map(applicationId -> generateAppHomeSC(applicationId))
+ .filter(appHomeSC -> judgeAdd(requestSC, appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC, result.size());
return GetApplicationsHomeSubClusterResponse.newInstance(result);
}
+ private ApplicationHomeSubCluster generateAppHomeSC(ApplicationId applicationId) {
+ SubClusterId subClusterId = applications.get(applicationId);
+ return ApplicationHomeSubCluster.newInstance(applicationId, subClusterId);
+ }
+
+ private boolean judgeAdd(SubClusterId filterSubCluster, SubClusterId homeSubCluster) {
+ if (filterSubCluster == null) {
+ return true;
+ } else if (filterSubCluster.equals(homeSubCluster)) {
Review Comment:
No need for else
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##########
@@ -949,7 +949,7 @@ public ReservationSubmissionResponse submitReservation(
// Second, determine whether the current ReservationId has a corresponding subCluster.
// If it does not exist, add it. If it exists, update it.
Boolean exists = existsReservationHomeSubCluster(reservationId);
- if (!exists) {
+ if (!exists || i == 0) {
Review Comment:
What is this for?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/GetApplicationsHomeSubClusterRequest.java:
##########
@@ -37,4 +38,33 @@ public static GetApplicationsHomeSubClusterRequest newInstance() {
return request;
}
+ @Private
+ @Unstable
+ public static GetApplicationsHomeSubClusterRequest
+ newInstance(SubClusterId subClusterId) {
+ GetApplicationsHomeSubClusterRequest request =
+ Records.newRecord(GetApplicationsHomeSubClusterRequest.class);
+ request.setSubClusterId(subClusterId);
+ return request;
+ }
+
+ /**
+ * Get the {@link SubClusterId} representing the unique identifier of the
+ * subcluster.
+ *
+ * @return the subcluster identifier
+ */
+ @InterfaceAudience.Public
Review Comment:
Use @Public
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1242477787
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 22s | | trunk passed |
| +1 :green_heart: | compile | 10m 47s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 25s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 32s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 37s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 43s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 8s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 35s | | the patch passed |
| +1 :green_heart: | compile | 9m 51s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 51s | | the patch passed |
| +1 :green_heart: | javac | 9m 51s | | the patch passed |
| +1 :green_heart: | compile | 9m 18s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 18s | | the patch passed |
| +1 :green_heart: | javac | 9m 18s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/15/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/15/artifact/out/blanks-tabs.txt) | The patch 10 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 1m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 57s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 17s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 237m 30s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 21s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 33s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 9s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not generate ASF License warnings. |
| | | 473m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 4fc5981d5f94 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c7620afccf192071761dbdbbfc97a5564649b8ef |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/15/testReport/ |
| Max. process+thread count | 2552 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1242502110
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 44s | | trunk passed |
| +1 :green_heart: | compile | 10m 38s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 9s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 10m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 46s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 6m 14s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 9s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 31s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 30s | | the patch passed |
| +1 :green_heart: | compile | 9m 56s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 56s | | the patch passed |
| +1 :green_heart: | javac | 9m 56s | | the patch passed |
| +1 :green_heart: | compile | 8m 59s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 59s | | the patch passed |
| +1 :green_heart: | javac | 8m 59s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/16/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/16/artifact/out/blanks-tabs.txt) | The patch 10 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 1m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 36s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 55s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 261m 23s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 21s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 7s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 24s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 5s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not generate ASF License warnings. |
| | | 503m 47s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 75e51bb99b0d 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c7620afccf192071761dbdbbfc97a5564649b8ef |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/16/testReport/ |
| Max. process+thread count | 1803 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238405786
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 25s | | trunk passed |
| +1 :green_heart: | compile | 10m 36s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 46s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 35s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 43s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 44s | | the patch passed |
| +1 :green_heart: | compile | 10m 19s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 10m 19s | | the patch passed |
| +1 :green_heart: | javac | 10m 19s | | the patch passed |
| +1 :green_heart: | compile | 9m 22s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 22s | | the patch passed |
| +1 :green_heart: | javac | 9m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 54s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 19s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 239m 38s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 34s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 33s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 32s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 12s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 21s | | The patch does not generate ASF License warnings. |
| | | 474m 49s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux d11f227539ba 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d309ac58589009db6f6a35a28a000d01b910fa3c |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/9/testReport/ |
| Max. process+thread count | 2479 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964252561
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##########
@@ -949,7 +949,7 @@ public ReservationSubmissionResponse submitReservation(
// Second, determine whether the current ReservationId has a corresponding subCluster.
// If it does not exist, add it. If it exists, update it.
Boolean exists = existsReservationHomeSubCluster(reservationId);
- if (!exists) {
+ if (!exists || i == 0) {
Review Comment:
Reservation may be repeatedly submitted (the same reservationId is submitted repeatedly). At this time, we should try to use the first reservation result. If the first reservation encounters some abnormalities, find another RM for reservation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r969997820
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,27 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM
+ (SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(partition by [homeSubCluster] order by [createTime] desc) AS row_num
+ FROM [dbo].[applicationsHomeSubCluster]) AS t
+ WHERE row_num <= @limit
+ AND (CASE WHEN @homeSubCluster IS NULL THEN 1
Review Comment:
I try to use `IF...ELSE` instead of `CASE...WHEN`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r962057774
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/SQLFederationStateStore.java:
##########
@@ -726,13 +731,23 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
CallableStatement cstmt = null;
ResultSet rs = null;
- List<ApplicationHomeSubCluster> appsHomeSubClusters =
- new ArrayList<ApplicationHomeSubCluster>();
+ List<ApplicationHomeSubCluster> appsHomeSubClusters = new ArrayList<>();
try {
cstmt = getCallableStatement(CALL_SP_GET_APPLICATIONS_HOME_SUBCLUSTER);
+ cstmt.setInt("limit_IN", maxAppsInStateStore);
+ String homeSubClusterIN = null;;
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r966545040
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,29 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ t4.applicationId,
+ t4.createTime,
+ t4.homeSubCluster,
+ t4.row_num
+ FROM (
+ SELECT
+ t2.applicationId,
+ t2.createTime,
+ t2.homeSubCluster,
+ @row_num := @row_num + 1 AS row_num
+ FROM (
+ SELECT
+ t.applicationId,
+ t.homeSubCluster,
+ t.createTime
+ FROM applicationshomesubcluster AS t
+ WHERE ( CASE WHEN t.homeSubCluster_IN IS NULL THEN 1 = 1 WHEN t.homeSubCluster_IN IS NOT NULL THEN homeSubCluster = homeSubCluster_IN END )
Review Comment:
I will continue to optimize the code and find an easy way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1247843639
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 33s | | trunk passed |
| +1 :green_heart: | compile | 10m 42s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 17s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 19s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 16s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 28s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 52s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 39s | | the patch passed |
| +1 :green_heart: | compile | 9m 18s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 18s | | the patch passed |
| +1 :green_heart: | javac | 9m 18s | | the patch passed |
| +1 :green_heart: | compile | 8m 46s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 46s | | the patch passed |
| +1 :green_heart: | javac | 8m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 11s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 38s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 237m 42s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/24/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 32s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 28s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 34s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 20s | | The patch does not generate ASF License warnings. |
| | | 464m 13s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/24/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 1095e20739b4 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 32042dce9b87ee5c582cd5243a7addd916060813 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/24/testReport/ |
| Max. process+thread count | 2477 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/24/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r962108067
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,14 +261,33 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
+ List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ List<ApplicationId> applicationIdList =
+ applications.keySet().stream().collect(Collectors.toList());
+
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
+ for (int i = 0; i < applicationIdList.size(); i++) {
+ if (appCount >= maxAppsInStateStore) {
Review Comment:
Thanks for your suggestion, I will modify the code!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r970137569
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,37 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ IF @homeSubCluster = ''
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM(SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(order by [createTime] desc) AS app_rank
+ FROM [dbo].[applicationsHomeSubCluster]) AS t
+ WHERE app_rank <= @limit;
+ ELSE
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM(SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(partition by [homeSubCluster] order by [createTime] desc) AS app_rank
+ FROM [dbo].[applicationsHomeSubCluster]
+ WHERE [homeSubCluster] = @homeSubCluster) AS t
Review Comment:
My code is a bit complicated, your approach is really good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r969998465
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,26 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ applicationId,
+ homeSubCluster,
+ createTime,
+ @app_rank := IF(@home_sc = homeSubCluster, @app_rank + 1, 1) AS app_rank,
+ @home_sc := homeSubCluster
+ FROM applicationshomesubcluster
+ ORDER BY createTime DESC
+ ) ranked
+ WHERE app_rank <= limit_IN
+ AND (CASE WHEN t.homeSubCluster_IN IS NULL THEN 1 = 1
Review Comment:
For MySQL, I also tried to refactor this SQL using a simpler way of writing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r971038757
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,24 +263,54 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- long start = clock.getTime();
- List<ApplicationHomeSubCluster> result = new ArrayList<>();
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
try {
- for (String child : zkManager.getChildren(appsZNode)) {
- ApplicationId appId = ApplicationId.fromString(child);
- SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
- result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List<String> children = zkManager.getChildren(appsZNode);
+ List<ApplicationHomeSubCluster> result =
+ children.stream().map(child -> generateAppHomeSC(child))
+ .sorted(Comparator.comparing(ApplicationHomeSubCluster::getCreateTime).reversed())
+ .filter(appHomeSC -> filterHomeSubCluster(requestSC, appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+ long end = clock.getTime();
+ opDurations.addGetAppsHomeSubClusterDuration(start, end);
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC, result.size());
+ return GetApplicationsHomeSubClusterResponse.newInstance(result);
} catch (Exception e) {
String errMsg = "Cannot get apps: " + e.getMessage();
FederationStateStoreUtils.logAndThrowStoreException(LOG, errMsg);
}
- long end = clock.getTime();
- opDurations.addGetAppsHomeSubClusterDuration(start, end);
- return GetApplicationsHomeSubClusterResponse.newInstance(result);
+
+ throw new YarnException("Cannot get app by request");
+ }
+
+ private ApplicationHomeSubCluster generateAppHomeSC(String appId) {
+ try {
+ ApplicationId applicationId = ApplicationId.fromString(appId);
+ SubClusterId homeSubCluster = getApp(applicationId);
+ ApplicationHomeSubCluster app =
+ ApplicationHomeSubCluster.newInstance(applicationId, Time.now(), homeSubCluster);
+ return app;
+ } catch (Exception ex) {
+ LOG.error("get homeSubCluster by appId = {}.", appId);
+ }
+ return null;
+ }
+
+ private boolean filterHomeSubCluster(SubClusterId filterSubCluster,
Review Comment:
We have this function defined three times in three places.
It should also be static.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,24 +263,54 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- long start = clock.getTime();
- List<ApplicationHomeSubCluster> result = new ArrayList<>();
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
try {
- for (String child : zkManager.getChildren(appsZNode)) {
- ApplicationId appId = ApplicationId.fromString(child);
- SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
- result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List<String> children = zkManager.getChildren(appsZNode);
+ List<ApplicationHomeSubCluster> result =
+ children.stream().map(child -> generateAppHomeSC(child))
+ .sorted(Comparator.comparing(ApplicationHomeSubCluster::getCreateTime).reversed())
Review Comment:
I don't think indentation is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238739439
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 22s | | trunk passed |
| +1 :green_heart: | compile | 10m 23s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 7s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 47s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 56s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 24s | | the patch passed |
| +1 :green_heart: | compile | 9m 16s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 16s | | the patch passed |
| +1 :green_heart: | javac | 9m 16s | | the patch passed |
| +1 :green_heart: | compile | 8m 35s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 35s | | the patch passed |
| +1 :green_heart: | javac | 8m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 46s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 9s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 58s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 34s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 22s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 29s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 27s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 17s | | The patch does not generate ASF License warnings. |
| | | 469m 17s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux d6d029dc8f61 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e96f7961e5198dcf4555fb468c9bf3d4969974b9 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/10/testReport/ |
| Max. process+thread count | 2773 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r961990524
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,14 +261,33 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
+ List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ List<ApplicationId> applicationIdList =
+ applications.keySet().stream().collect(Collectors.toList());
+
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
+ for (int i = 0; i < applicationIdList.size(); i++) {
+ if (appCount >= maxAppsInStateStore) {
Review Comment:
Move this to the for condition.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,23 +260,41 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
long start = clock.getTime();
List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
try {
- for (String child : zkManager.getChildren(appsZNode)) {
+ List<String> childrens = zkManager.getChildren(appsZNode);
+ for (String child : childrens) {
+ if (appCount >= maxAppsInStateStore) {
+ break;
+ }
ApplicationId appId = ApplicationId.fromString(child);
SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
+ // If the requestSubClusterId that needs to be filtered in the request
+ // is inconsistent with the SubClusterId in the data, continue to the next round
+ if (requestSubClusterId != null && !requestSubClusterId.equals(homeSubCluster)) {
+ continue;
Review Comment:
reverse the if
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/SQLFederationStateStore.java:
##########
@@ -726,13 +731,23 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
CallableStatement cstmt = null;
ResultSet rs = null;
- List<ApplicationHomeSubCluster> appsHomeSubClusters =
- new ArrayList<ApplicationHomeSubCluster>();
+ List<ApplicationHomeSubCluster> appsHomeSubClusters = new ArrayList<>();
try {
cstmt = getCallableStatement(CALL_SP_GET_APPLICATIONS_HOME_SUBCLUSTER);
+ cstmt.setInt("limit_IN", maxAppsInStateStore);
+ String homeSubClusterIN = null;;
Review Comment:
Extra ;
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,14 +261,33 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
+ List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ List<ApplicationId> applicationIdList =
+ applications.keySet().stream().collect(Collectors.toList());
+
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
+ for (int i = 0; i < applicationIdList.size(); i++) {
+ if (appCount >= maxAppsInStateStore) {
+ break;
+ }
+ ApplicationId applicationId = applicationIdList.get(i);
+ SubClusterId subClusterId = applications.get(applicationId);
+ // If the requestSubClusterId that needs to be filtered in the request
+ // is inconsistent with the SubClusterId in the data, continue to the next round
+ if (requestSubClusterId != null && !requestSubClusterId.equals(subClusterId)){
+ continue;
+ }
+ result.add(ApplicationHomeSubCluster.newInstance(applicationId, subClusterId));
+ appCount++;
}
- GetApplicationsHomeSubClusterResponse.newInstance(result);
Review Comment:
This was just overlooked?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,14 +261,33 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
+ List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ List<ApplicationId> applicationIdList =
+ applications.keySet().stream().collect(Collectors.toList());
+
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
+ for (int i = 0; i < applicationIdList.size(); i++) {
+ if (appCount >= maxAppsInStateStore) {
Review Comment:
Actually it would be good to do a foreach
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,14 +261,33 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
+ List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ List<ApplicationId> applicationIdList =
+ applications.keySet().stream().collect(Collectors.toList());
+
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
+ for (int i = 0; i < applicationIdList.size(); i++) {
+ if (appCount >= maxAppsInStateStore) {
+ break;
+ }
+ ApplicationId applicationId = applicationIdList.get(i);
+ SubClusterId subClusterId = applications.get(applicationId);
+ // If the requestSubClusterId that needs to be filtered in the request
+ // is inconsistent with the SubClusterId in the data, continue to the next round
+ if (requestSubClusterId != null && !requestSubClusterId.equals(subClusterId)){
+ continue;
+ }
+ result.add(ApplicationHomeSubCluster.newInstance(applicationId, subClusterId));
Review Comment:
do the reverse if to avoid the continue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1243013741
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 24s | | trunk passed |
| +1 :green_heart: | compile | 9m 54s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 45s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 26s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 13s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 37s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 35s | | the patch passed |
| +1 :green_heart: | compile | 9m 16s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 16s | | the patch passed |
| +1 :green_heart: | javac | 9m 16s | | the patch passed |
| +1 :green_heart: | compile | 8m 31s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 31s | | the patch passed |
| +1 :green_heart: | javac | 8m 31s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/17/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/17/artifact/out/blanks-tabs.txt) | The patch 10 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 1m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 37s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 59s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 9s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 212m 10s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/17/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch failed. |
| +1 :green_heart: | unit | 1m 41s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 21s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 27s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 4s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 35s | | The patch does not generate ASF License warnings. |
| | | 438m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux dd7d592ba7dd 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 358977b4fe67657968086e6544e8d93ccf7f9b6e |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/17/testReport/ |
| Max. process+thread count | 2772 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1243941828
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 35s | | trunk passed |
| +1 :green_heart: | compile | 10m 22s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 19s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 6s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 14s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 3s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 25s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 16s | | the patch passed |
| +1 :green_heart: | compile | 10m 2s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 10m 2s | | the patch passed |
| +1 :green_heart: | javac | 10m 2s | | the patch passed |
| +1 :green_heart: | compile | 9m 13s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 13s | | the patch passed |
| +1 :green_heart: | javac | 9m 13s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/18/artifact/out/blanks-tabs.txt) | The patch 10 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 1m 47s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 40s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 51s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 55s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 212m 34s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/18/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch failed. |
| +1 :green_heart: | unit | 1m 34s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 22s | | hadoop-yarn-common in the patch passed. |
| -1 :x: | unit | 3m 39s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/18/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not generate ASF License warnings. |
| | | 438m 10s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 905eeba1a63c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a536e30b188accfc6024ae18c4fa5a4707a7cd96 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/18/testReport/ |
| Max. process+thread count | 2479 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r972541463
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,24 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM(SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(order by [createTime] desc) AS app_rank
+ FROM [dbo].[applicationsHomeSubCluster]
+ WHERE [homeSubCluster] = @homeSubCluster OR @homeSubCluster = '') AS t
Review Comment:
Thank you very much for helping to review the code, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1237793763
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 27s | | trunk passed |
| +1 :green_heart: | compile | 9m 53s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 39s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 29s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 42s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 24s | | the patch passed |
| +1 :green_heart: | compile | 9m 15s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 15s | | the patch passed |
| +1 :green_heart: | javac | 9m 15s | | the patch passed |
| +1 :green_heart: | compile | 8m 36s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 36s | | the patch passed |
| +1 :green_heart: | javac | 8m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 50s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 165 unchanged - 0 fixed = 168 total (was 165) |
| +1 :green_heart: | mvnsite | 8m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 50s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 9s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 41s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 31s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 40s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 35s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 10s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 22s | | The patch does not generate ASF License warnings. |
| | | 468m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux e336fb1ca805 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e1ee2a3a5968f91d217c5597537c6ee1b253c21d |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/7/testReport/ |
| Max. process+thread count | 2552 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238215896
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 40s | | trunk passed |
| +1 :green_heart: | compile | 10m 12s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 3s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 43s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 39s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 24s | | the patch passed |
| +1 :green_heart: | compile | 9m 16s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 16s | | the patch passed |
| +1 :green_heart: | javac | 9m 16s | | the patch passed |
| +1 :green_heart: | compile | 8m 54s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 54s | | the patch passed |
| +1 :green_heart: | javac | 8m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 54s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 165 unchanged - 0 fixed = 168 total (was 165) |
| +1 :green_heart: | mvnsite | 8m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 49s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 20s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 30s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 33s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 28s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 35s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 24s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 23s | | The patch does not generate ASF License warnings. |
| | | 472m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux b9d14515684f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 023bb5c6fa2b255957ee493b5fccf602fe5e4cd2 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/8/testReport/ |
| Max. process+thread count | 2552 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238308788
@goiri Please help to review the code again, thank you very much!
This PR changes the execution script of the stored procedure, I will check the script against the major version of the database.
**SQLServer**
The change script of the stored procedure has been verified on SQL Server 2012 Enterprise And SQL Server 2008 R2 Enterprise, I will verify it on the following version as soon as possible
- [ ] SQL Server 2008 R2 Enterprise
- [ ] SQL Server 2012 Enterprise
SQL Server 2016 Enterprise
SQL Server 2017 Enterprise
SQL Server 2019 Enterprise
**MySQL**
MySQL 5.7 and above must be required, MySQL 5.5 and MySQL 5.6 cannot be supported.
MySQL-5.5 and MySQL-5.6 Can't Create Table membership and policies, for the following reasons:
1.The primary key of the membership table is subClusterId varchar(256) , Mysql will prompt the following error:
Specified key was too long; max key length is 767 bytes.
2.The primary key of the policies table is subClusterId varchar(256) , Mysql will prompt the following error:
Specified key was too long; max key length is 767 bytes.
- [ ] MySQL 5.7
- [ ] MySQL 8.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964252561
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##########
@@ -949,7 +949,7 @@ public ReservationSubmissionResponse submitReservation(
// Second, determine whether the current ReservationId has a corresponding subCluster.
// If it does not exist, add it. If it exists, update it.
Boolean exists = existsReservationHomeSubCluster(reservationId);
- if (!exists) {
+ if (!exists || i == 0) {
Review Comment:
Reservation may be repeatedly submitted (the same reservationId is submitted repeatedly). At this time, we should try to use the first reservation result. If the first reservation encounters some abnormalities, find another RM for reservation.
If a reservationId(**reservationId_1**) has successfully applied for reserved resources in SC_1, and the **reservationId_1** is submitted repeatedly, we should try to use the allocation result of SC_1
For the case of **reservationId_1**, the following code will return true,
```
Boolean exists = existsReservationHomeSubCluster(reservationId);
```
Then the first subcluster(SC_1) will be used instead of SC2
```
updateReservationHomeSubCluster(subClusterId, reservationId, reservationHomeSubCluster);
```
But at this time we still want to go `addReservationHomeSubCluster(reservationId, reservationHomeSubCluster)`
`addReservationHomeSubCluster`, will not be added repeatedly for the existing reservationId.
I added a condition, if i=0, continue to use `addReservationHomeSubCluster`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964250673
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/GetApplicationsHomeSubClusterRequest.java:
##########
@@ -37,4 +38,33 @@ public static GetApplicationsHomeSubClusterRequest newInstance() {
return request;
}
+ @Private
+ @Unstable
+ public static GetApplicationsHomeSubClusterRequest
+ newInstance(SubClusterId subClusterId) {
+ GetApplicationsHomeSubClusterRequest request =
+ Records.newRecord(GetApplicationsHomeSubClusterRequest.class);
+ request.setSubClusterId(subClusterId);
+ return request;
+ }
+
+ /**
+ * Get the {@link SubClusterId} representing the unique identifier of the
+ * subcluster.
+ *
+ * @return the subcluster identifier
+ */
+ @InterfaceAudience.Public
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r962108019
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,14 +261,33 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
+ List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ List<ApplicationId> applicationIdList =
+ applications.keySet().stream().collect(Collectors.toList());
+
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
+ for (int i = 0; i < applicationIdList.size(); i++) {
+ if (appCount >= maxAppsInStateStore) {
+ break;
+ }
+ ApplicationId applicationId = applicationIdList.get(i);
+ SubClusterId subClusterId = applications.get(applicationId);
+ // If the requestSubClusterId that needs to be filtered in the request
+ // is inconsistent with the SubClusterId in the data, continue to the next round
+ if (requestSubClusterId != null && !requestSubClusterId.equals(subClusterId)){
+ continue;
+ }
+ result.add(ApplicationHomeSubCluster.newInstance(applicationId, subClusterId));
+ appCount++;
}
- GetApplicationsHomeSubClusterResponse.newInstance(result);
Review Comment:
This line of code should be an extra line of code, which has no practical significance. The following is directly constructed and returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r962114313
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,14 +261,33 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
+
+ List<ApplicationHomeSubCluster> result = new ArrayList<>();
+ List<ApplicationId> applicationIdList =
+ applications.keySet().stream().collect(Collectors.toList());
+
+ SubClusterId requestSubClusterId = request.getSubClusterId();
+ int appCount = 0;
+ for (int i = 0; i < applicationIdList.size(); i++) {
+ if (appCount >= maxAppsInStateStore) {
+ break;
+ }
+ ApplicationId applicationId = applicationIdList.get(i);
+ SubClusterId subClusterId = applications.get(applicationId);
+ // If the requestSubClusterId that needs to be filtered in the request
+ // is inconsistent with the SubClusterId in the data, continue to the next round
+ if (requestSubClusterId != null && !requestSubClusterId.equals(subClusterId)){
+ continue;
+ }
+ result.add(ApplicationHomeSubCluster.newInstance(applicationId, subClusterId));
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1236421286
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 4s | | trunk passed |
| +1 :green_heart: | compile | 10m 26s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 0s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 48s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 0s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 4s | | the patch passed |
| +1 :green_heart: | compile | 9m 15s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 15s | | the patch passed |
| +1 :green_heart: | javac | 9m 15s | | the patch passed |
| +1 :green_heart: | compile | 8m 35s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 35s | | the patch passed |
| +1 :green_heart: | javac | 8m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 47s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 16s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 35s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 238m 7s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 26s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 26s | | The patch does not generate ASF License warnings. |
| | | 458m 39s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 0a1b88b41cf6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9c499ca83293ef4d8074c5e9aed8b14b8cd0b5ad |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/3/testReport/ |
| Max. process+thread count | 2774 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1236511195
This PR changes the execution script of the stored procedure, I will check the script against the major version of the database.
SQL Server
The change script of the stored procedure has been verified on SQL Server 2012 Enterprise, I will verify it on the following version as soon as possible
- SQL Server 2008 R2 Enterprise
- SQL Server 2016 Enterprise
- SQL Server 2017 Enterprise
- SQL Server 2019 Enterprise
MySQL
- MySQL 5.5
- MySQL 5.6
- MySQL 5.7
- MySQL 8.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r970000124
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/FederationStateStoreBaseTest.java:
##########
@@ -410,6 +414,89 @@ public void testGetApplicationsHomeSubCluster() throws Exception {
Assert.assertTrue(result.getAppsHomeSubClusters().contains(ahsc2));
}
+ @Test
+ public void testGetApplicationsHomeSubClusterEmpty() throws Exception {
+ LambdaTestUtils.intercept(YarnException.class,
+ "Missing getApplicationsHomeSubCluster request",
+ () -> stateStore.getApplicationsHomeSubCluster(null));
+ }
+
+ @Test
+ public void testGetApplicationsHomeSubClusterFilter() throws Exception {
+ // Add ApplicationHomeSC - SC1
+ long now = Time.now();
+
+ Set<ApplicationHomeSubCluster> appHomeSubClusters = new HashSet<>();
+
+ for (int i = 0; i < TEN_ROUNDS; i++) {
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1245869268
@goiri Please help to review the code again, thank you very much!
I have provided a test report, we can refer to the following link
https://issues.apache.org/jira/secure/attachment/13049175/Improve%20Query%20Condition%20of%20FederationStateStore%23getApplicationsHomeSubCluster%20Test%20Report.pdf
MySQL
We need 5.7 and above
- [ ] MySQL 5.7
- [ ] MySQL 8.0
SQLServer
- [ ] SQL Server 2008 R2 Enterprise
- [ ] SQL Server 2012 Enterprise
- [ ] SQL Server 2016 Enterprise
- [ ] SQL Server 2017 Enterprise
- [ ] SQL Server 2019 Enterprise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r969999747
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/ApplicationHomeSubCluster.java:
##########
@@ -51,6 +51,18 @@ public static ApplicationHomeSubCluster newInstance(ApplicationId appId,
return appMapping;
}
+ @Private
+ @Unstable
+ public static ApplicationHomeSubCluster newInstance(ApplicationId appId, long createTime,
+ SubClusterId homeSubCluster) {
+ ApplicationHomeSubCluster appMapping =
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1242351273
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 41s | | trunk passed |
| +1 :green_heart: | compile | 9m 57s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 47s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 40s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 59s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 3s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 27s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 35s | | the patch passed |
| +1 :green_heart: | compile | 9m 21s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 21s | | the patch passed |
| +1 :green_heart: | javac | 9m 21s | | the patch passed |
| +1 :green_heart: | compile | 9m 4s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 4s | | the patch passed |
| +1 :green_heart: | javac | 9m 4s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/14/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 1m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 38s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 57s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 20m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 14s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 30s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 29s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 32s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 13s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not generate ASF License warnings. |
| | | 469m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 3e6d4a64c4f7 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2a09fa7802cb3da50e11f47884cdd9cbaeafdc4e |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/14/testReport/ |
| Max. process+thread count | 2700 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1256079979
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 48s | | trunk passed |
| +1 :green_heart: | compile | 11m 10s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 10m 24s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 21s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 34s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 16s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 27s | | the patch passed |
| +1 :green_heart: | compile | 10m 51s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 10m 51s | | the patch passed |
| +1 :green_heart: | javac | 10m 51s | | the patch passed |
| +1 :green_heart: | compile | 9m 45s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 45s | | the patch passed |
| +1 :green_heart: | javac | 9m 45s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/28/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 56s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/28/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 165 unchanged - 0 fixed = 167 total (was 165) |
| +1 :green_heart: | mvnsite | 8m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 46s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 56s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 22s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 34s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 45s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not generate ASF License warnings. |
| | | 477m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/28/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 4ed2274fafc0 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 27be2559844006e804884fdb75c3f261d82e51aa |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/28/testReport/ |
| Max. process+thread count | 2775 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/28/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r965313989
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,27 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM
+ (SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(partition by [homeSubCluster] order by [createTime] desc) as row_num
Review Comment:
Yes, we should get TopN's app, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r965333105
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,17 +261,45 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
}
- GetApplicationsHomeSubClusterResponse.newInstance(result);
+ SubClusterId requestSC = request.getSubClusterId();
+ List<ApplicationHomeSubCluster> result = applications.keySet().stream()
Review Comment:
I will add sorting logic.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,24 +261,53 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- long start = clock.getTime();
- List<ApplicationHomeSubCluster> result = new ArrayList<>();
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
try {
- for (String child : zkManager.getChildren(appsZNode)) {
- ApplicationId appId = ApplicationId.fromString(child);
- SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
- result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List<String> children = zkManager.getChildren(appsZNode);
+ List<ApplicationHomeSubCluster> result =
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1241339614
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 34s | | trunk passed |
| +1 :green_heart: | compile | 9m 47s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 48s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 28s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 53s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 35s | | the patch passed |
| +1 :green_heart: | compile | 9m 15s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 15s | | the patch passed |
| +1 :green_heart: | javac | 9m 15s | | the patch passed |
| +1 :green_heart: | compile | 8m 39s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 39s | | the patch passed |
| +1 :green_heart: | javac | 8m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 36s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 9s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 28s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 240m 49s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 31s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 27s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 21s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 3m 57s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not generate ASF License warnings. |
| | | 469m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 39ecd88331c4 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f6df901f99a27a82e70e4b1aefbdc0b7cfdfd11c |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/13/testReport/ |
| Max. process+thread count | 2774 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r966535217
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,29 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ t4.applicationId,
+ t4.createTime,
+ t4.homeSubCluster,
+ t4.row_num
+ FROM (
+ SELECT
+ t2.applicationId,
+ t2.createTime,
+ t2.homeSubCluster,
+ @row_num := @row_num + 1 AS row_num
+ FROM (
+ SELECT
+ t.applicationId,
+ t.homeSubCluster,
+ t.createTime
+ FROM applicationshomesubcluster AS t
+ WHERE ( CASE WHEN t.homeSubCluster_IN IS NULL THEN 1 = 1 WHEN t.homeSubCluster_IN IS NOT NULL THEN homeSubCluster = homeSubCluster_IN END )
Review Comment:
Wow! Is this the cleanest way to return the top N records?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri merged pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri merged PR #4846:
URL: https://github.com/apache/hadoop/pull/4846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964250393
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,17 +261,36 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
}
- GetApplicationsHomeSubClusterResponse.newInstance(result);
+ SubClusterId requestSC = request.getSubClusterId();
+ List<ApplicationHomeSubCluster> result = applications.keySet().stream()
+ .map(applicationId -> generateAppHomeSC(applicationId))
+ .filter(appHomeSC -> judgeAdd(requestSC, appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC, result.size());
return GetApplicationsHomeSubClusterResponse.newInstance(result);
}
+ private ApplicationHomeSubCluster generateAppHomeSC(ApplicationId applicationId) {
+ SubClusterId subClusterId = applications.get(applicationId);
+ return ApplicationHomeSubCluster.newInstance(applicationId, subClusterId);
+ }
+
+ private boolean judgeAdd(SubClusterId filterSubCluster, SubClusterId homeSubCluster) {
+ if (filterSubCluster == null) {
+ return true;
+ } else if (filterSubCluster.equals(homeSubCluster)) {
Review Comment:
Thanks for your help reviewing the code, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1239142291
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 8s | | trunk passed |
| +1 :green_heart: | compile | 10m 53s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 30s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 45s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 52s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 23s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 51s | | the patch passed |
| +1 :green_heart: | compile | 10m 10s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 10m 10s | | the patch passed |
| +1 :green_heart: | javac | 10m 10s | | the patch passed |
| +1 :green_heart: | compile | 9m 16s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 16s | | the patch passed |
| +1 :green_heart: | javac | 9m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 3s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 20s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 7s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 240m 25s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 42s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 22s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | unit | 4m 27s | | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 21s | | The patch does not generate ASF License warnings. |
| | | 479m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 36ea07a14f0b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 40f6d999ba00a9881ea94bdb374ec4441fa7fdd6 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/12/testReport/ |
| Max. process+thread count | 2552 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r970137060
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,21 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ *,
+ @rownum := 0,
+ IF(homeSubCluster_IN = '', 1, (homeSubCluster = homeSubCluster_IN)) AS filter_result
+ FROM applicationshomesubcluster
+ ORDER BY createTime DESC) AS app_home_sc
+ WHERE filter_result = 1
Review Comment:
Thank you very much for your suggestion, I agree with you, I will modify the code!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1246173571
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 24m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 46s | | trunk passed |
| +1 :green_heart: | compile | 9m 54s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 48s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 50s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 14s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 35s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 59s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 0s | | the patch passed |
| +1 :green_heart: | compile | 9m 12s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 12s | | the patch passed |
| +1 :green_heart: | javac | 9m 12s | | the patch passed |
| +1 :green_heart: | compile | 8m 43s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 43s | | the patch passed |
| +1 :green_heart: | javac | 8m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 5s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 36s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 239m 38s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/21/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 33s | | hadoop-yarn-common in the patch passed. |
| -1 :x: | unit | 3m 39s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/21/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not generate ASF License warnings. |
| | | 466m 16s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 328350126d92 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a12ea7e34e1260902736341242523cb44d31a303 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/21/testReport/ |
| Max. process+thread count | 2552 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/21/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1248591269
@goiri Please help to review the code again, thank you very much!
Error in TestLogAggregationService in unit test, I have submitted pr separately to fix.
(https://github.com/apache/hadoop/pull/4893)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1235812702
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 35s | | trunk passed |
| +1 :green_heart: | compile | 9m 51s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 38s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 50s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 17s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 15m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 41s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 6s | | the patch passed |
| +1 :green_heart: | compile | 9m 9s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 9s | | the patch passed |
| +1 :green_heart: | javac | 9m 9s | | the patch passed |
| +1 :green_heart: | compile | 8m 30s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 30s | | the patch passed |
| +1 :green_heart: | javac | 8m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 46s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 165 unchanged - 0 fixed = 171 total (was 165) |
| +1 :green_heart: | mvnsite | 6m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 19s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 45s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 16m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 16s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 238m 45s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| -1 :x: | unit | 1m 33s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt) | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 3m 28s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 15s | | The patch does not generate ASF License warnings. |
| | | 442m 18s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| | hadoop.yarn.conf.TestYarnConfigurationFields |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle cc buflint bufcompat |
| uname | Linux 13a9c47fd080 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ba324a542449d5092621dda960d2959e9cf9458a |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/1/testReport/ |
| Max. process+thread count | 2774 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1236140992
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 32s | | trunk passed |
| +1 :green_heart: | compile | 9m 51s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 52s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 11s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 36s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 53s | | the patch passed |
| +1 :green_heart: | compile | 9m 13s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 13s | | the patch passed |
| +1 :green_heart: | javac | 9m 13s | | the patch passed |
| +1 :green_heart: | compile | 8m 38s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 38s | | the patch passed |
| +1 :green_heart: | javac | 8m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 49s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 165 unchanged - 0 fixed = 168 total (was 165) |
| +1 :green_heart: | mvnsite | 7m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 13s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 42s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 12m 2s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/2/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn.html) | hadoop-yarn-project/hadoop-yarn generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| -1 :x: | spotbugs | 1m 34s | [/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/2/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 20m 43s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 237m 32s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 38s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 31s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 30s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 38s | | The patch does not generate ASF License warnings. |
| | | 454m 42s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn |
| | Redundant nullcheck of filterSubCluster, which is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at MemoryFederationStateStore.java:is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at MemoryFederationStateStore.java:[line 288] |
| | Redundant nullcheck of filterSubCluster, which is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at ZookeeperFederationStateStore.java:is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at ZookeeperFederationStateStore.java:[line 306] |
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| | Redundant nullcheck of filterSubCluster, which is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at MemoryFederationStateStore.java:is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at MemoryFederationStateStore.java:[line 288] |
| | Redundant nullcheck of filterSubCluster, which is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at ZookeeperFederationStateStore.java:is known to be non-null in org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore.judgeAdd(SubClusterId, SubClusterId) Redundant null check at ZookeeperFederationStateStore.java:[line 306] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux da8b0101bd89 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a1629b7b74d2f77035dee07e0a9cf4477ebe6d07 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/2/testReport/ |
| Max. process+thread count | 2775 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1259032015
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 52s | | trunk passed |
| +1 :green_heart: | compile | 9m 56s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 44s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 49s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 6m 11s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 11s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 37s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 3s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 13s | | the patch passed |
| +1 :green_heart: | compile | 9m 13s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 13s | | the patch passed |
| +1 :green_heart: | javac | 9m 13s | | the patch passed |
| +1 :green_heart: | compile | 8m 35s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 35s | | the patch passed |
| +1 :green_heart: | javac | 8m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 10s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 34s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 16s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 40s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 24s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 47s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not generate ASF License warnings. |
| | | 459m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/30/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 3eecc188ca4c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 794f79bd5cb65c4fb60d2e93bb380bca6fef8e25 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/30/testReport/ |
| Max. process+thread count | 2478 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/30/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1256603495
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 35s | | trunk passed |
| +1 :green_heart: | compile | 9m 58s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 50s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 43s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 2s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 5s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 32s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 57s | | the patch passed |
| +1 :green_heart: | compile | 9m 14s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 14s | | the patch passed |
| +1 :green_heart: | javac | 9m 14s | | the patch passed |
| +1 :green_heart: | compile | 8m 37s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 37s | | the patch passed |
| +1 :green_heart: | javac | 8m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 46s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 57s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 56s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 14s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 22s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 35s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not generate ASF License warnings. |
| | | 457m 38s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/29/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux bb6866d35151 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / da64b19149b280412b01e0450b17fa2865f5fd3f |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/29/testReport/ |
| Max. process+thread count | 2700 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/29/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1260032750
@goiri Thank you very much for your help reviewing the code!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r971434340
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,24 +263,54 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- long start = clock.getTime();
- List<ApplicationHomeSubCluster> result = new ArrayList<>();
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
try {
- for (String child : zkManager.getChildren(appsZNode)) {
- ApplicationId appId = ApplicationId.fromString(child);
- SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
- result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List<String> children = zkManager.getChildren(appsZNode);
+ List<ApplicationHomeSubCluster> result =
+ children.stream().map(child -> generateAppHomeSC(child))
+ .sorted(Comparator.comparing(ApplicationHomeSubCluster::getCreateTime).reversed())
Review Comment:
Thanks for your suggestion, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1236593039
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 40s | | trunk passed |
| +1 :green_heart: | compile | 9m 53s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 49s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 54s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 4s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 46s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 46s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 3s | | the patch passed |
| +1 :green_heart: | compile | 9m 14s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 14s | | the patch passed |
| +1 :green_heart: | javac | 9m 14s | | the patch passed |
| +1 :green_heart: | compile | 8m 38s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 38s | | the patch passed |
| +1 :green_heart: | javac | 8m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 7m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 10s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 32s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 17m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 238m 42s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 38s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 29s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 36s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 20s | | The patch does not generate ASF License warnings. |
| | | 456m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux 28b5bc65f642 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 354b8b3ba130fe90e92cfddaddce8e9ebfa3fa44 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/4/testReport/ |
| Max. process+thread count | 2479 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1237189620
@goiri Please help to review the code again, thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1237512502
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 7 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 38s | | trunk passed |
| +1 :green_heart: | compile | 9m 56s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 42s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 15s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 43s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 3s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 40s | | the patch passed |
| +1 :green_heart: | compile | 9m 7s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 7s | | the patch passed |
| +1 :green_heart: | javac | 9m 7s | | the patch passed |
| +1 :green_heart: | compile | 8m 38s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 38s | | the patch passed |
| +1 :green_heart: | javac | 8m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 47s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 165 unchanged - 0 fixed = 168 total (was 165) |
| +1 :green_heart: | mvnsite | 8m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 44s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 14s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 237m 48s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/6/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 32s | | hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 30s | | hadoop-yarn-common in the patch passed. |
| +1 :green_heart: | unit | 3m 39s | | hadoop-yarn-server-common in the patch passed. |
| -1 :x: | unit | 4m 27s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/6/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt) | hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 16s | | The patch does not generate ASF License warnings. |
| | | 464m 38s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor |
| | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc buflint bufcompat |
| uname | Linux a8cff8289f40 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 499986a60fe4964123963da2b54b817c79f2c2ca |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/6/testReport/ |
| Max. process+thread count | 2776 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r965083185
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreStoreProcs.sql:
##########
@@ -111,12 +111,27 @@ IF OBJECT_ID ( '[sp_getApplicationsHomeSubCluster]', 'P' ) IS NOT NULL
GO
CREATE PROCEDURE [dbo].[sp_getApplicationsHomeSubCluster]
+ @limit int,
+ @homeSubCluster VARCHAR(256)
AS BEGIN
DECLARE @errorMessage nvarchar(4000)
BEGIN TRY
- SELECT [applicationId], [homeSubCluster], [createTime]
- FROM [dbo].[applicationsHomeSubCluster]
+ SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime]
+ FROM
+ (SELECT
+ [applicationId],
+ [homeSubCluster],
+ [createTime],
+ row_number() over(partition by [homeSubCluster] order by [createTime] desc) as row_num
Review Comment:
We are getting the top N newest reservations right?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##########
@@ -255,17 +261,45 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- List<ApplicationHomeSubCluster> result =
- new ArrayList<ApplicationHomeSubCluster>();
- for (Entry<ApplicationId, SubClusterId> e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(), e.getValue()));
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
}
- GetApplicationsHomeSubClusterResponse.newInstance(result);
+ SubClusterId requestSC = request.getSubClusterId();
+ List<ApplicationHomeSubCluster> result = applications.keySet().stream()
Review Comment:
Do we have any order here?
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,17 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SET @row_num = 0;
+ SELECT
+ applicationId, homeSubCluster
+ FROM (SELECT
+ applicationId, homeSubCluster, (@rownum := @rownum + 1) AS row_num
+ FROM applicationsHomeSubCluster) AS t
+ WHERE row_num <= limit_IN
Review Comment:
Given that we don't order, aren't we getting random apps?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##########
@@ -255,24 +261,53 @@ public GetApplicationHomeSubClusterResponse getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
- long start = clock.getTime();
- List<ApplicationHomeSubCluster> result = new ArrayList<>();
+
+ if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+ }
try {
- for (String child : zkManager.getChildren(appsZNode)) {
- ApplicationId appId = ApplicationId.fromString(child);
- SubClusterId homeSubCluster = getApp(appId);
- ApplicationHomeSubCluster app =
- ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
- result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List<String> children = zkManager.getChildren(appsZNode);
+ List<ApplicationHomeSubCluster> result =
Review Comment:
Order?
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml:
##########
@@ -4963,4 +4963,14 @@
</description>
</property>
+ <property>
+ <name>yarn.federation.state-store.max-applications</name>
+ <value>1000</value>
+ <description>
+ yarn federation state-store supports
Review Comment:
Fix the capitalization while making it one line and add periods at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r965313622
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,17 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SET @row_num = 0;
+ SELECT
+ applicationId, homeSubCluster
+ FROM (SELECT
+ applicationId, homeSubCluster, (@rownum := @rownum + 1) AS row_num
+ FROM applicationsHomeSubCluster) AS t
+ WHERE row_num <= limit_IN
Review Comment:
Thank you very much for your suggestion, we should sort the apps and get the topN apps, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r965316896
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml:
##########
@@ -4963,4 +4963,14 @@
</description>
</property>
+ <property>
+ <name>yarn.federation.state-store.max-applications</name>
+ <value>1000</value>
+ <description>
+ yarn federation state-store supports
Review Comment:
Thanks for your suggestion, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4846: YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster.
Posted by GitBox <gi...@apache.org>.
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r969064754
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -122,10 +122,26 @@ BEGIN
WHERE applicationId = applicationID_IN;
END //
-CREATE PROCEDURE sp_getApplicationsHomeSubCluster()
-BEGIN
- SELECT applicationId, homeSubCluster
- FROM applicationsHomeSubCluster;
+CREATE PROCEDURE sp_getApplicationsHomeSubCluster(IN limit_IN int, IN homeSubCluster_IN varchar(256))
+BEGIN
+ SELECT
+ applicationId,
+ homeSubCluster,
+ createTime
+ FROM
+ (SELECT
+ applicationId,
+ homeSubCluster,
+ createTime,
+ @app_rank := IF(@home_sc = homeSubCluster, @app_rank + 1, 1) AS app_rank,
+ @home_sc := homeSubCluster
+ FROM applicationshomesubcluster
Review Comment:
Thanks for your help reviewing the code, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org