You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/08/25 04:41:25 UTC
[GitHub] [hadoop-ozone] amaliujia opened a new pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
amaliujia opened a new pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346
## What changes were proposed in this pull request?
CLI command to show current SCM leader and follower status. E.g. `ozone admin scmha listratisstatus`
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4115
## How was this patch tested?
Unit Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495538752
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestScmAdminHA.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell;
+
+import java.net.InetSocketAddress;
+import java.util.UUID;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.admin.OzoneAdmin;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * This class tests ozone admin scm commands.
+ */
+public class TestScmAdminHA {
+ private static OzoneAdmin ozoneAdmin;
+ private static OzoneConfiguration conf;
+ private static String omServiceId;
+ private static int numOfOMs;
+ private static String clusterId;
+ private static String scmId;
+ private static MiniOzoneCluster cluster;
+
+ @BeforeClass
+ public static void init() throws Exception {
+ ozoneAdmin = new OzoneAdmin();
+ conf = new OzoneConfiguration();
+
+ // Init HA cluster
+ omServiceId = "om-service-test1";
Review comment:
You mean SCM here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-689826469
R @timmylicheng @nandakumar131
I am thinking maybe we can first merge this PR and create a JIRA to track left work. Right now per feedback this command could print more information about Ratis peers, e.g. leader/follower roles, leader term, etc.
I took a look at how does OM HA does: https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L2535
Basically it seems such request will directly hit the leader OM, then get status will be much easier. Currently in SCM HA we haven't reached to the point with robust Ratis setup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481136753
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##########
@@ -560,6 +563,13 @@ public boolean getReplicationManagerStatus() {
return scm.getReplicationManager().isRunning();
}
+ @Override
+ public List<String> getScmRatisStatus() throws IOException {
+ return scm.getScmHAManager()
+ .getRatisServer().getRaftPeers()
+ .stream().map(peer -> peer.getAddress()).collect(Collectors.toList());
Review comment:
I can see this method be wrapper in ScmHAManager to show a list of scm hosts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481638257
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
I am ok with both.
@adoroszlai what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481142517
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
Name seems a bit verbose. Let me find a syntax to align with OM HA
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481380452
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
`status` or `roles` would probably be enough to indicate the goal of the subcommand, something like
```
ozone admin (om|scm) status
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-681793105
Could you attach your CLI command output?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-685173208
Addressed the following comments
1. Merge `getRatisStatus` with `GetScmInfo`
2. Adopt command syntax `ozone admin scm status`
3. added an acceptance test
@timmylicheng I am not sure how to test an acceptance test. Can you share a way to run it locally?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495539471
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -107,6 +108,14 @@ public void shutdown() throws IOException {
ratisServer.stop();
}
+ @Override
+ public List<String> getRatisStatus() {
+ return Arrays.asList(
Review comment:
You can use the default SCM RatisServer port. Check ratisBindPort in SCMHAConfiguration
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481469131
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
##########
@@ -230,4 +230,5 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type,
*/
boolean getReplicationManagerStatus() throws IOException;
+ List<String> getScmRatisStatus() throws IOException;
Review comment:
Pushed a new commit to merge this logic into `GetScmInfo`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481469412
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestScmAdminHA.java
##########
@@ -0,0 +1,66 @@
+package org.apache.hadoop.ozone.shell;
Review comment:
ack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481359383
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
I think both this one and OM HA `getserviceroles` can be improved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia edited a comment on pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia edited a comment on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-679615937
R: @timmylicheng
This is still WIP. Can you give some suggestions? Just want to make sure I am on the right track.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481134741
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java
##########
@@ -271,6 +273,14 @@ public ScmContainerLocationResponse processRequest(
.setGetSafeModeRuleStatusesResponse(getSafeModeRuleStatues(
request.getGetSafeModeRuleStatusesRequest()))
.build();
+ case ScmHAStatus:
Review comment:
GetSCMRatisRole will be better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-699599648
Thanks @amaliujia for the contribution. The patch looks good overall. Just a few comments inline.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495601341
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestScmAdminHA.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.shell;
+
+import java.net.InetSocketAddress;
+import java.util.UUID;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.admin.OzoneAdmin;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * This class tests ozone admin scm commands.
+ */
+public class TestScmAdminHA {
+ private static OzoneAdmin ozoneAdmin;
+ private static OzoneConfiguration conf;
+ private static String omServiceId;
+ private static int numOfOMs;
+ private static String clusterId;
+ private static String scmId;
+ private static MiniOzoneCluster cluster;
+
+ @BeforeClass
+ public static void init() throws Exception {
+ ozoneAdmin = new OzoneAdmin();
+ conf = new OzoneConfiguration();
+
+ // Init HA cluster
+ omServiceId = "om-service-test1";
Review comment:
Actually I meant OM. It turns out that I cannot ignore `omServiceId` to start the cluster (there is a check for this service id).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-700441836
Re @bshashikant
Agreed. Right now the command itself is unified (for both OM and SCM, we name this command as `roles`). Then we should unify the behavior fo both commands (and if there is DN command, that should be the same).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant edited a comment on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
bshashikant edited a comment on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-700435789
I feel the CLI should be common for both OM and SCM and probably extended to Datanodes as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481135941
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##########
@@ -560,6 +563,13 @@ public boolean getReplicationManagerStatus() {
return scm.getReplicationManager().isRunning();
}
+ @Override
+ public List<String> getScmRatisStatus() throws IOException {
+ return scm.getScmHAManager()
+ .getRatisServer().getRaftPeers()
+ .stream().map(peer -> peer.getAddress()).collect(Collectors.toList());
Review comment:
Could potentially throw NPE
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-682325031
Ok I think I can work to have an example output first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-689140982
Also R: @nandakumar131 can you please take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-682120312
@timmylicheng
The output is empty array. In the test setup, `scm.getScmHAManager().getRatisServer().getRaftPeers();` return empty list. My current guess is I haven't enabled SCM HA in the test setup.
I expect it will print something like `[127.0.1.1:1313, 127.0.1.1:1314, 127.0.1.1:1315]`. So I guess this JIRA requires a format of, e.g., `[leader: 127.0.1.1:1313, follower: 127.0.1.1:1314, follower: 127.0.1.1:1315]`?
So if my approach looks reasonable, I can continue working on this version to have a valid test somehow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481146665
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestScmAdminHA.java
##########
@@ -0,0 +1,66 @@
+package org.apache.hadoop.ozone.shell;
Review comment:
Apart from UT, we can have an acceptance test to test CLI.
You may find examples here: https://github.com/apache/hadoop-ozone/pull/375
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
##########
@@ -230,4 +230,5 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type,
*/
boolean getReplicationManagerStatus() throws IOException;
+ List<String> getScmRatisStatus() throws IOException;
Review comment:
Any way to merge it with GetSCMInfo?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-679615937
R: @timmylicheng
This is still WIP. Can you give some suggestions. Just want to make sure I am on the right track.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481414123
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
Thanks. I will adopt `ozone admin scm status` in this PR and I will send another PR for `ozone admin om status`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-700491675
+1. Thanks for Rui's contribution.
Merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481371486
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
@adoroszlai
What kind of improvement OM HA `getserviceroles` you are thinking of? I am happy to make a separate PR for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r477533235
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##########
@@ -560,6 +563,13 @@ public boolean getReplicationManagerStatus() {
return scm.getReplicationManager().isRunning();
}
+ @Override
+ public List<String> getScmRatisStatus() throws IOException {
+ return scm.getScmHAManager()
+ .getRatisServer().getRaftPeers()
+ .stream().map(peer -> peer.getAddress()).collect(Collectors.toList());
Review comment:
One thing I am not sure is what should be fetched from Ratis peers. Should I also include which one is leader and which one is follower?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-699688323
@timmylicheng comments addressed. Can you take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495538644
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/ScmAdmin.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.admin.scm;
+
+import org.apache.hadoop.hdds.cli.GenericCli;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.ozone.admin.OzoneAdmin;
+import picocli.CommandLine;
+import picocli.CommandLine.Model.CommandSpec;
+import picocli.CommandLine.Spec;
+
+/**
+ * Subcommand for admin operations related to SCM.
+ */
+@CommandLine.Command(
Review comment:
why do we need a separate scmAdmin?
Shall we have like ozone admin getSCMRoles?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495601428
##########
File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java
##########
@@ -107,6 +108,14 @@ public void shutdown() throws IOException {
ratisServer.stop();
}
+ @Override
+ public List<String> getRatisStatus() {
+ return Arrays.asList(
Review comment:
Switched to use the default port. In the future I will think about how to better construct `MockSCMHAManager` for testing purpose (as the code is evolving we can keep updating `MockSCMHAManager`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-683243594
@timmylicheng
I found the reason why in my current test empty array was printed. It is because the test is using `MockSCMHaManager`: https://github.com/apache/hadoop-ozone/blob/HDDS-2823/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java#L191
I manually add a peer locally and get the output print: `[localhost:7238]`, which is expected.
Any suggestion how should I proceed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-685425652
Uploaded one commit to
1. use `ozone admin scm roles`
2. fix the acceptance test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481557217
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
ozone admin om getserviceroles -id=<<service id>>
This is what OM does. I have my +1 on ozone admin (om|scm) roles. Status is more like health check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481414123
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
Thanks. I will adopt `ozone admin scm status` in this PR and I will send another PR for ``ozone admin om status`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481746270
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
that makes sense. Indeed `status` sounds more like a health check and can carry much more information.
Consider `OM` is also actually getting `roles`. We can start from `roles`
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
that makes sense. Indeed `status` sounds more like a health check and can carry much more information.
Consider `OM` is also actually getting, essentially, `roles`. We can start from `roles`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-700435789
The CLI should be generic for both OM and SCM and probably extended to Datanodes as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia edited a comment on pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia edited a comment on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-682120312
@timmylicheng
The output is empty array. In current test setup, `scm.getScmHAManager().getRatisServer().getRaftPeers()` returns empty list. My current guess is I haven't enabled SCM HA in the test setup in a correct way.
I expect it will print something like `[127.0.1.1:1313, 127.0.1.1:1314, 127.0.1.1:1315]`. Also I guess this JIRA requires a format of, e.g., `[leader: 127.0.1.1:1313, follower: 127.0.1.1:1314, follower: 127.0.1.1:1315]`?
So if my approach looks reasonable, I can continue working on this version to have a valid test somehow to make sure it prints desired formats .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495601298
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+/**
+ * Handler of scm status command.
+ */
+@CommandLine.Command(
+ name = "roles",
+ description = "List all SCMs and their respective Ratis server status",
+ mixinStandardHelpOptions = true,
+ versionProvider = HddsVersionProvider.class)
+public class GetScmRatisStatusSubcommand implements Callable<Void> {
+
+ @CommandLine.ParentCommand
+ private ScmAdmin parent;
+
+ @Override
+ public Void call() throws Exception {
+ ScmClient scmClient = parent.createScmClient();
+ List<String> status = scmClient.getScmRatisStatus();
Review comment:
Good point. I have switched all `status` to `roles`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng merged pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng merged pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481709338
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,26 @@
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+@CommandLine.Command(
+ name = "listratisstatus",
Review comment:
I think it depends on whether you want to keep this command specific to roles, or may extend the same command in the future with other status info. Probably "roles" is better now, and "status" can be either a separate command or another alias later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r481378454
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##########
@@ -560,6 +563,13 @@ public boolean getReplicationManagerStatus() {
return scm.getReplicationManager().isRunning();
}
+ @Override
+ public List<String> getScmRatisStatus() throws IOException {
+ return scm.getScmHAManager()
+ .getRatisServer().getRaftPeers()
+ .stream().map(peer -> peer.getAddress()).collect(Collectors.toList());
Review comment:
Got it. Will move this functionality to ScmHAManager.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r477534266
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestScmAdminHA.java
##########
@@ -0,0 +1,66 @@
+package org.apache.hadoop.ozone.shell;
+
+import java.io.File;
+import java.net.InetSocketAddress;
+import java.util.Arrays;
+import java.util.UUID;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.admin.OzoneAdmin;
+import org.apache.hadoop.ozone.admin.scm.ScmAdmin;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import picocli.CommandLine;
+import picocli.CommandLine.RunLast;
+
+public class TestScmAdminHA {
+ private static OzoneAdmin ozoneAdmin;
+ private static OzoneConfiguration conf;
+ private static String omServiceId;
+ private static int numOfOMs;
+ private static String clusterId;
+ private static String scmId;
+ private static MiniOzoneCluster cluster;
+
+ @BeforeClass
+ public static void init() throws Exception {
+ ozoneAdmin = new OzoneAdmin();
+ conf = new OzoneConfiguration();
+
+ // Init HA cluster
+ omServiceId = "om-service-test1";
+ numOfOMs = 3;
+ clusterId = UUID.randomUUID().toString();
+ scmId = UUID.randomUUID().toString();
+ cluster = MiniOzoneCluster.newHABuilder(conf)
+ .setClusterId(clusterId)
+ .setScmId(scmId)
+ .setOMServiceId(omServiceId)
+ .setNumOfOzoneManagers(numOfOMs)
+ .build();
+ conf.setQuietMode(false);
+ // enable ratis for Scm.
+ conf.setBoolean(ScmConfigKeys.DFS_CONTAINER_RATIS_ENABLED_KEY, true);
+ cluster.waitForClusterToBeReady();
+ }
+
+ @AfterClass
+ public static void shutdown() {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Test
+ public void testGetLeaderStatus() {
+ InetSocketAddress address = cluster.getStorageContainerManager().getClientRpcAddress();
+ String hostPort = address.getHostName() + ":" + address.getPort();
+ String[] args = {"--scm", hostPort, "scmha", "listratisstatus"};
+ ozoneAdmin.execute(args);
Review comment:
This is actually not a unit test as it does not check what is returned. I will think about whether there is a simple way to test that API directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1346: [WIP] HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r477533780
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
##########
@@ -230,4 +230,5 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type,
*/
boolean getReplicationManagerStatus() throws IOException;
+ List<String> getScmRatisStatus() throws IOException;
Review comment:
Does this API belongs to `StorageContainerLocationProtocol`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495538535
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.admin.scm;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+/**
+ * Handler of scm status command.
+ */
+@CommandLine.Command(
+ name = "roles",
+ description = "List all SCMs and their respective Ratis server status",
+ mixinStandardHelpOptions = true,
+ versionProvider = HddsVersionProvider.class)
+public class GetScmRatisStatusSubcommand implements Callable<Void> {
+
+ @CommandLine.ParentCommand
+ private ScmAdmin parent;
+
+ @Override
+ public Void call() throws Exception {
+ ScmClient scmClient = parent.createScmClient();
+ List<String> status = scmClient.getScmRatisStatus();
Review comment:
We probably want to align the naming here. Either we use roles or status for all external and internal interfaces.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] amaliujia edited a comment on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.
Posted by GitBox <gi...@apache.org>.
amaliujia edited a comment on pull request #1346:
URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-689826469
R @timmylicheng @nandakumar131
I am thinking maybe we can first merge this PR and create a JIRA to track left work. Right now per feedback this command could print more information about Ratis peers, e.g. leader/follower roles, leader term, etc.
I took a look at how does OM HA does: https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L2535
Basically it seems such request will directly hit the leader OM, then get status will be much easier. Currently in SCM HA we haven't reached to the point with a robust Ratis setup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org