You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/01/10 18:48:08 UTC
[GitHub] [ozone] aswinshakil opened a new pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
aswinshakil opened a new pull request #2972:
URL: https://github.com/apache/ozone/pull/2972
## What changes were proposed in this pull request?
When Recon startup, Based on a threshold value we need to get SCM DB using gRPC and populate the Recon DB in SCM HA enabled cluster.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6135
## How was this patch tested?
This patch was tested manually and using integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a change in pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on a change in pull request #2972:
URL: https://github.com/apache/ozone/pull/2972#discussion_r781484934
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconScmHASnapshot.java
##########
@@ -0,0 +1,136 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon;
+
+import org.apache.hadoop.hdds.client.RatisReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.container.ContainerInfo;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineManager;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.ozone.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test Recon SCM HA Snapshot Download implementation.
+ */
+public class TestReconScmHASnapshot {
+
+ /**
+ * Set a timeout for each test.
+ */
+ @Rule
+ public Timeout timeout = Timeout.seconds(100);
+ private static OzoneConfiguration conf;
+ private static MiniOzoneCluster cluster;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ conf = new OzoneConfiguration();
+ conf.setBoolean(OZONE_SCM_HA_ENABLE_KEY, true);
+ conf.setBoolean(
+ ReconServerConfigKeys.OZONE_RECON_SCM_SNAPSHOT_ENABLED, true);
+ conf.setInt(ReconServerConfigKeys.OZONE_RECON_SCM_CONTAINER_THRESHOLD, 0);
+ conf.setInt(ScmConfigKeys.OZONE_DATANODE_PIPELINE_LIMIT, 5);
+ cluster = MiniOzoneCluster.newBuilder(conf)
+ .setNumDatanodes(4)
+ .includeRecon(true)
+ .build();
+ cluster.waitForClusterToBeReady();
+ }
+
+ @Test
+ public void testScmHASnapshot() throws Exception {
+ GenericTestUtils.LogCapturer logCapturer = GenericTestUtils.LogCapturer
+ .captureLogs(LoggerFactory.getLogger(
+ ReconStorageContainerManagerFacade.class));
+
+ List<ContainerInfo> reconContainers = cluster.getReconServer()
+ .getReconStorageContainerManager().getContainerManager()
+ .getContainers();
+ assertEquals(0, reconContainers.size());
+
+ ReconNodeManager nodeManager;
+ nodeManager = (ReconNodeManager) cluster.getReconServer()
+ .getReconStorageContainerManager().getScmNodeManager();
+ long keyCountBefore = nodeManager.getNodeDBKeyCount();
+
+ //Stopping Recon to add Containers in SCM
+ cluster.stopRecon();
+
+ ContainerManager containerManager;
+ containerManager = cluster.getStorageContainerManager()
+ .getContainerManager();
+
+ for (int i = 0; i < 10; i++) {
+ containerManager.allocateContainer(new RatisReplicationConfig(
+ HddsProtos.ReplicationFactor.ONE), "testOwner");
+ }
+
+ cluster.startRecon();
+
+ //ContainerCount after Recon DB is updated with SCM DB
+ containerManager = cluster.getStorageContainerManager()
+ .getContainerManager();
+ ContainerManager reconContainerManager = cluster.getReconServer()
+ .getReconStorageContainerManager().getContainerManager();
+ assertTrue(logCapturer.getOutput()
+ .contains("Downloaded SCM Snapshot from Leader SCM"));
+ assertTrue(logCapturer.getOutput()
+ .contains("Recon Container Count: " + reconContainers.size() +
+ ", SCM Container Count: " + containerManager.getContainers().size()));
+ assertEquals(containerManager.getContainers().size(),
+ reconContainerManager.getContainers().size());
+
+ //PipelineCount after Recon DB is updated with SCM DB
+ PipelineManager scmPipelineManager = cluster.getStorageContainerManager()
+ .getPipelineManager();
+ PipelineManager reconPipelineManager = cluster.getReconServer()
+ .getReconStorageContainerManager().getPipelineManager();
+ assertEquals(scmPipelineManager.getPipelines().size(),
+ reconPipelineManager.getPipelines().size());
+
+ //NodeCount after Recon DB updated with SCM DB
+ nodeManager = (ReconNodeManager) cluster.getReconServer()
+ .getReconStorageContainerManager().getScmNodeManager();
+ long keyCountAfter = nodeManager.getNodeDBKeyCount();
+ assertEquals(keyCountAfter, keyCountBefore);
Review comment:
Can we extract this into a common assertion function shared between the HA and non HA variants of this test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aswinshakil commented on a change in pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
aswinshakil commented on a change in pull request #2972:
URL: https://github.com/apache/ozone/pull/2972#discussion_r786992332
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java
##########
@@ -179,4 +185,38 @@ public DBCheckpoint getSCMDBSnapshot() {
}
return null;
}
+
+ public DBCheckpoint getSCMDBSnapshotHA() throws IOException {
Review comment:
Sure will do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a change in pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #2972:
URL: https://github.com/apache/ozone/pull/2972#discussion_r786897992
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java
##########
@@ -179,4 +185,38 @@ public DBCheckpoint getSCMDBSnapshot() {
}
return null;
}
+
+ public DBCheckpoint getSCMDBSnapshotHA() throws IOException {
Review comment:
Refactor this new method and `getSCMDBSnapshot()` to reduce code duplication.
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java
##########
@@ -349,7 +350,13 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ DBCheckpoint dbSnapshot;
+ if(!SCMHAUtils.isSCMHAEnabled(ozoneConfiguration)) {
Review comment:
nit
```suggestion
if (!SCMHAUtils.isSCMHAEnabled(ozoneConfiguration)) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a change in pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on a change in pull request #2972:
URL: https://github.com/apache/ozone/pull/2972#discussion_r781483317
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconScmHASnapshot.java
##########
@@ -0,0 +1,136 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon;
+
+import org.apache.hadoop.hdds.client.RatisReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.container.ContainerInfo;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineManager;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.ozone.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test Recon SCM HA Snapshot Download implementation.
+ */
+public class TestReconScmHASnapshot {
+
+ /**
+ * Set a timeout for each test.
+ */
+ @Rule
+ public Timeout timeout = Timeout.seconds(100);
+ private static OzoneConfiguration conf;
+ private static MiniOzoneCluster cluster;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ conf = new OzoneConfiguration();
+ conf.setBoolean(OZONE_SCM_HA_ENABLE_KEY, true);
+ conf.setBoolean(
+ ReconServerConfigKeys.OZONE_RECON_SCM_SNAPSHOT_ENABLED, true);
+ conf.setInt(ReconServerConfigKeys.OZONE_RECON_SCM_CONTAINER_THRESHOLD, 0);
+ conf.setInt(ScmConfigKeys.OZONE_DATANODE_PIPELINE_LIMIT, 5);
+ cluster = MiniOzoneCluster.newBuilder(conf)
+ .setNumDatanodes(4)
+ .includeRecon(true)
+ .build();
+ cluster.waitForClusterToBeReady();
+ }
+
+ @Test
+ public void testScmHASnapshot() throws Exception {
+ GenericTestUtils.LogCapturer logCapturer = GenericTestUtils.LogCapturer
+ .captureLogs(LoggerFactory.getLogger(
+ ReconStorageContainerManagerFacade.class));
+
+ List<ContainerInfo> reconContainers = cluster.getReconServer()
+ .getReconStorageContainerManager().getContainerManager()
+ .getContainers();
+ assertEquals(0, reconContainers.size());
+
+ ReconNodeManager nodeManager;
+ nodeManager = (ReconNodeManager) cluster.getReconServer()
+ .getReconStorageContainerManager().getScmNodeManager();
+ long keyCountBefore = nodeManager.getNodeDBKeyCount();
+
+ //Stopping Recon to add Containers in SCM
+ cluster.stopRecon();
+
+ ContainerManager containerManager;
+ containerManager = cluster.getStorageContainerManager()
+ .getContainerManager();
+
+ for (int i = 0; i < 10; i++) {
+ containerManager.allocateContainer(new RatisReplicationConfig(
+ HddsProtos.ReplicationFactor.ONE), "testOwner");
+ }
+
+ cluster.startRecon();
Review comment:
Can we move this to a common initializer function shared between the `TestReconScmHASnapshot.java` and `TestReconScmSnapshot.java`, it will help reduce code duplication across the two files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aswinshakil commented on a change in pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
aswinshakil commented on a change in pull request #2972:
URL: https://github.com/apache/ozone/pull/2972#discussion_r788183703
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java
##########
@@ -160,15 +167,39 @@ public DBCheckpoint getSCMDBSnapshot() {
".tar.gz");
try {
- SecurityUtil.doAsLoginUser(() -> {
- try (InputStream inputStream = reconUtils.makeHttpCall(
- connectionFactory, getScmDBSnapshotUrl(),
- isOmSpnegoEnabled()).getInputStream()) {
- FileUtils.copyInputStreamToFile(inputStream, targetFile);
+ if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+ SecurityUtil.doAsLoginUser(() -> {
+ try (InputStream inputStream = reconUtils.makeHttpCall(
+ connectionFactory, getScmDBSnapshotUrl(),
+ isOmSpnegoEnabled()).getInputStream()) {
+ FileUtils.copyInputStreamToFile(inputStream, targetFile);
+ }
+ return null;
+ });
+ } else {
+ List<String> ratisRoles = scmClient.getScmInfo().getRatisPeerRoles();
+ for (String ratisRole: ratisRoles) {
+ String[] role = ratisRole.split(":");
+ if(role[2].equals(RaftProtos.RaftPeerRole.LEADER.toString())) {
+ String hostAddress = role[4].trim();
+ int grpcPort = configuration.getInt(
+ ScmConfigKeys.OZONE_SCM_GRPC_PORT_KEY,
+ ScmConfigKeys.OZONE_SCM_GRPC_PORT_DEFAULT);
+
+ try (SCMSnapshotDownloader downloadClient =
+ new InterSCMGrpcClient(hostAddress, grpcPort,
+ configuration, new SCMCertificateClient(
+ new SecurityConfig(configuration)))) {
+ downloadClient.download(targetFile.toPath()).get();
+ } catch (ExecutionException | InterruptedException e) {
+ LOG.error("Rocks DB checkpoint downloading failed", e);
+ throw new IOException(e);
+ }
+ LOG.info("Downloaded SCM Snapshot from Leader SCM");
Review comment:
Updated the PR with the suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl merged pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
smengcl merged pull request #2972:
URL: https://github.com/apache/ozone/pull/2972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on pull request #2972:
URL: https://github.com/apache/ozone/pull/2972#issuecomment-1009269880
Thank you for the contribution @aswinshakil! I have left a few minor comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a change in pull request #2972: HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.
Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #2972:
URL: https://github.com/apache/ozone/pull/2972#discussion_r788155134
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java
##########
@@ -160,15 +167,39 @@ public DBCheckpoint getSCMDBSnapshot() {
".tar.gz");
try {
- SecurityUtil.doAsLoginUser(() -> {
- try (InputStream inputStream = reconUtils.makeHttpCall(
- connectionFactory, getScmDBSnapshotUrl(),
- isOmSpnegoEnabled()).getInputStream()) {
- FileUtils.copyInputStreamToFile(inputStream, targetFile);
+ if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+ SecurityUtil.doAsLoginUser(() -> {
+ try (InputStream inputStream = reconUtils.makeHttpCall(
+ connectionFactory, getScmDBSnapshotUrl(),
+ isOmSpnegoEnabled()).getInputStream()) {
+ FileUtils.copyInputStreamToFile(inputStream, targetFile);
+ }
+ return null;
+ });
+ } else {
+ List<String> ratisRoles = scmClient.getScmInfo().getRatisPeerRoles();
+ for (String ratisRole: ratisRoles) {
+ String[] role = ratisRole.split(":");
+ if(role[2].equals(RaftProtos.RaftPeerRole.LEADER.toString())) {
+ String hostAddress = role[4].trim();
+ int grpcPort = configuration.getInt(
+ ScmConfigKeys.OZONE_SCM_GRPC_PORT_KEY,
+ ScmConfigKeys.OZONE_SCM_GRPC_PORT_DEFAULT);
+
+ try (SCMSnapshotDownloader downloadClient =
+ new InterSCMGrpcClient(hostAddress, grpcPort,
+ configuration, new SCMCertificateClient(
+ new SecurityConfig(configuration)))) {
+ downloadClient.download(targetFile.toPath()).get();
+ } catch (ExecutionException | InterruptedException e) {
+ LOG.error("Rocks DB checkpoint downloading failed", e);
+ throw new IOException(e);
+ }
+ LOG.info("Downloaded SCM Snapshot from Leader SCM");
Review comment:
Let's add a simalar message to non-HA code path as well.
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java
##########
@@ -160,15 +167,39 @@ public DBCheckpoint getSCMDBSnapshot() {
".tar.gz");
try {
- SecurityUtil.doAsLoginUser(() -> {
- try (InputStream inputStream = reconUtils.makeHttpCall(
- connectionFactory, getScmDBSnapshotUrl(),
- isOmSpnegoEnabled()).getInputStream()) {
- FileUtils.copyInputStreamToFile(inputStream, targetFile);
+ if (!SCMHAUtils.isSCMHAEnabled(configuration)) {
+ SecurityUtil.doAsLoginUser(() -> {
+ try (InputStream inputStream = reconUtils.makeHttpCall(
+ connectionFactory, getScmDBSnapshotUrl(),
+ isOmSpnegoEnabled()).getInputStream()) {
+ FileUtils.copyInputStreamToFile(inputStream, targetFile);
+ }
+ return null;
+ });
+ } else {
+ List<String> ratisRoles = scmClient.getScmInfo().getRatisPeerRoles();
+ for (String ratisRole: ratisRoles) {
+ String[] role = ratisRole.split(":");
+ if(role[2].equals(RaftProtos.RaftPeerRole.LEADER.toString())) {
Review comment:
```suggestion
if (role[2].equals(RaftProtos.RaftPeerRole.LEADER.toString())) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org