You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "vtutrinov (via GitHub)" <gi...@apache.org> on 2024/11/22 09:40:59 UTC

[PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

vtutrinov opened a new pull request, #7471:
URL: https://github.com/apache/ozone/pull/7471

   ## What changes were proposed in this pull request?
   Reset SCM leadership metric to null only in case the leader is not defined earlier on initializing the RAFT server
   
   When the SCM servers are in HA mode and start simultaneously leadership manages to settle down before the HTTP server starts. But the SCM main thread set the leadership to null with the suggestion the leadership is not defined yet, and the leadership will not be defined on prometheus metrics level:
   
   ```bash
   bash-5.1$ for scm in {scm1,scm2,scm3}; do echo ${scm} && curl -s http://${scm}:9876/prom | grep scmha_metrics_scmha_leader_state ; done
   scm1
   # TYPE scmha_metrics_scmha_leader_state gauge
   scmha_metrics_scmha_leader_state{nodeid="a8b30f45-58b0-4ab2-b52b-9aae232ccc10",hostname="e8121b3ddde2"} 0
   scm2
   # TYPE scmha_metrics_scmha_leader_state gauge
   scmha_metrics_scmha_leader_state{nodeid="ae525b25-2774-4f91-9bbc-83884c1f472d",hostname="a467ff8d65d7"} 0
   scm3
   # TYPE scmha_metrics_scmha_leader_state gauge
   scmha_metrics_scmha_leader_state{nodeid="a5d83fc8-7fe3-4d49-b8f6-159e411bf0fa",hostname="2ea45cb6b00a"} 0
   ```
   The PR suggests resetting the leadership metric to null only in case the leader is not defined earlier on initializing the RAFT server. Also, the HA mini-cluster was rewritten to imitate the simultaneous start of the SCM servers in HA mode
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-11742
   
   ## How was this patch tested?
   With existing integration tests after making changes for the Ozone HA mini-cluster
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. Update metrics with leaderId if known when starting SCM [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #7471:
URL: https://github.com/apache/ozone/pull/7471


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1862994084


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -323,4 +335,75 @@ private void waitForLeaderToBeReady()
     }, 1000, (int) ScmConfigKeys
         .OZONE_SCM_HA_RATIS_LEADER_READY_WAIT_TIMEOUT_DEFAULT);
   }
+
+  @Test
+  public void testSCMLeadershipMetric() throws IOException, InterruptedException {
+    // GIVEN
+    int scmInstancesCount = 3;
+    conf = new OzoneConfiguration();
+    MiniOzoneCluster.Builder haMiniClusterBuilder = MiniOzoneCluster.newHABuilder(conf)
+        .setSCMServiceId("scm-service-id")
+        .setOMServiceId("om-service-id")
+        .setNumOfActiveOMs(0)
+        .setNumOfStorageContainerManagers(scmInstancesCount)
+        .setNumOfActiveSCMs(1)
+        .setNumDatanodes(0);
+
+    // start single SCM instance without other Ozone services
+    // in order to initialize and bootstrap SCM instances only
+    cluster = (MiniOzoneHAClusterImpl) haMiniClusterBuilder.build();
+
+    List<StorageContainerManager> storageContainerManagersList =
+        ((MiniOzoneHAClusterImpl) cluster).getStorageContainerManagersList();
+
+    // stop the single SCM instance in order to imitate further simultaneous start of SCMs
+    storageContainerManagersList.get(0).stop();
+    storageContainerManagersList.get(0).join();
+
+    // WHEN (imitate simultaneous start of the SCMs)
+    CountDownLatch scmInstancesCounter = new CountDownLatch(scmInstancesCount);
+    int retryCount = 0;
+    while (true) {
+      AtomicInteger failedSCMs = new AtomicInteger();

Review Comment:
   Done



##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -323,4 +335,75 @@ private void waitForLeaderToBeReady()
     }, 1000, (int) ScmConfigKeys
         .OZONE_SCM_HA_RATIS_LEADER_READY_WAIT_TIMEOUT_DEFAULT);
   }
+
+  @Test
+  public void testSCMLeadershipMetric() throws IOException, InterruptedException {
+    // GIVEN
+    int scmInstancesCount = 3;
+    conf = new OzoneConfiguration();
+    MiniOzoneCluster.Builder haMiniClusterBuilder = MiniOzoneCluster.newHABuilder(conf)
+        .setSCMServiceId("scm-service-id")
+        .setOMServiceId("om-service-id")
+        .setNumOfActiveOMs(0)
+        .setNumOfStorageContainerManagers(scmInstancesCount)
+        .setNumOfActiveSCMs(1)
+        .setNumDatanodes(0);
+
+    // start single SCM instance without other Ozone services
+    // in order to initialize and bootstrap SCM instances only
+    cluster = (MiniOzoneHAClusterImpl) haMiniClusterBuilder.build();
+
+    List<StorageContainerManager> storageContainerManagersList =
+        ((MiniOzoneHAClusterImpl) cluster).getStorageContainerManagersList();
+
+    // stop the single SCM instance in order to imitate further simultaneous start of SCMs
+    storageContainerManagersList.get(0).stop();
+    storageContainerManagersList.get(0).join();
+
+    // WHEN (imitate simultaneous start of the SCMs)
+    CountDownLatch scmInstancesCounter = new CountDownLatch(scmInstancesCount);
+    int retryCount = 0;
+    while (true) {
+      AtomicInteger failedSCMs = new AtomicInteger();
+      for (StorageContainerManager scm : storageContainerManagersList) {
+        CountDownLatch finalScmInstancesCounter = scmInstancesCounter;
+        new Thread(() -> {
+          try {
+            scm.start();
+          } catch (IOException e) {
+            failedSCMs.incrementAndGet();
+          } finally {
+            finalScmInstancesCounter.countDown();
+          }
+        }).start();
+      }
+      scmInstancesCounter.await();
+      if (failedSCMs.get() == 0) {
+        break;
+      } else {
+        for (StorageContainerManager scm : storageContainerManagersList) {
+          scm.stop();
+          scm.join();
+          LOG.info("Stopping StorageContainerManager server at {}",
+              scm.getClientRpcAddress());
+        }
+        ++retryCount;
+        LOG.info("SCMs port conflicts, retried {} times",
+            retryCount);
+        failedSCMs.set(0);
+        scmInstancesCounter = new CountDownLatch(scmInstancesCount);
+      }
+    }
+
+    // THEN expect at least one SCM node (leader) will have 'scmha_metrics_scmha_leader_state' metric set to 1

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1862972717


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -323,4 +335,75 @@ private void waitForLeaderToBeReady()
     }, 1000, (int) ScmConfigKeys
         .OZONE_SCM_HA_RATIS_LEADER_READY_WAIT_TIMEOUT_DEFAULT);
   }
+
+  @Test
+  public void testSCMLeadershipMetric() throws IOException, InterruptedException {
+    // GIVEN
+    int scmInstancesCount = 3;
+    conf = new OzoneConfiguration();
+    MiniOzoneCluster.Builder haMiniClusterBuilder = MiniOzoneCluster.newHABuilder(conf)
+        .setSCMServiceId("scm-service-id")
+        .setOMServiceId("om-service-id")
+        .setNumOfActiveOMs(0)
+        .setNumOfStorageContainerManagers(scmInstancesCount)
+        .setNumOfActiveSCMs(1)
+        .setNumDatanodes(0);
+
+    // start single SCM instance without other Ozone services
+    // in order to initialize and bootstrap SCM instances only
+    cluster = (MiniOzoneHAClusterImpl) haMiniClusterBuilder.build();
+
+    List<StorageContainerManager> storageContainerManagersList =
+        ((MiniOzoneHAClusterImpl) cluster).getStorageContainerManagersList();
+
+    // stop the single SCM instance in order to imitate further simultaneous start of SCMs
+    storageContainerManagersList.get(0).stop();
+    storageContainerManagersList.get(0).join();
+
+    // WHEN (imitate simultaneous start of the SCMs)
+    CountDownLatch scmInstancesCounter = new CountDownLatch(scmInstancesCount);
+    int retryCount = 0;
+    while (true) {
+      AtomicInteger failedSCMs = new AtomicInteger();
+      for (StorageContainerManager scm : storageContainerManagersList) {
+        CountDownLatch finalScmInstancesCounter = scmInstancesCounter;
+        new Thread(() -> {
+          try {
+            scm.start();
+          } catch (IOException e) {
+            failedSCMs.incrementAndGet();
+          } finally {
+            finalScmInstancesCounter.countDown();
+          }
+        }).start();
+      }
+      scmInstancesCounter.await();
+      if (failedSCMs.get() == 0) {
+        break;
+      } else {
+        for (StorageContainerManager scm : storageContainerManagersList) {
+          scm.stop();
+          scm.join();
+          LOG.info("Stopping StorageContainerManager server at {}",
+              scm.getClientRpcAddress());
+        }
+        ++retryCount;
+        LOG.info("SCMs port conflicts, retried {} times",
+            retryCount);
+        failedSCMs.set(0);
+        scmInstancesCounter = new CountDownLatch(scmInstancesCount);
+      }
+    }
+
+    // THEN expect at least one SCM node (leader) will have 'scmha_metrics_scmha_leader_state' metric set to 1

Review Comment:
   Is more than one leader OK?  If not, we can assert that the sum of metrics is 1.



##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -72,6 +77,8 @@
 @Timeout(300)
 public class TestStorageContainerManagerHA {
 
+  private static final Logger LOG = LoggerFactory.getLogger(TestMiniOzoneCluster.class);

Review Comment:
   ```suggestion
     private static final Logger LOG = LoggerFactory.getLogger(TestStorageContainerManagerHA.class);
   ```



##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -323,4 +335,75 @@ private void waitForLeaderToBeReady()
     }, 1000, (int) ScmConfigKeys
         .OZONE_SCM_HA_RATIS_LEADER_READY_WAIT_TIMEOUT_DEFAULT);
   }
+
+  @Test
+  public void testSCMLeadershipMetric() throws IOException, InterruptedException {
+    // GIVEN
+    int scmInstancesCount = 3;
+    conf = new OzoneConfiguration();
+    MiniOzoneCluster.Builder haMiniClusterBuilder = MiniOzoneCluster.newHABuilder(conf)
+        .setSCMServiceId("scm-service-id")
+        .setOMServiceId("om-service-id")
+        .setNumOfActiveOMs(0)
+        .setNumOfStorageContainerManagers(scmInstancesCount)
+        .setNumOfActiveSCMs(1)
+        .setNumDatanodes(0);
+
+    // start single SCM instance without other Ozone services
+    // in order to initialize and bootstrap SCM instances only
+    cluster = (MiniOzoneHAClusterImpl) haMiniClusterBuilder.build();
+
+    List<StorageContainerManager> storageContainerManagersList =
+        ((MiniOzoneHAClusterImpl) cluster).getStorageContainerManagersList();

Review Comment:
   We can avoid the need to cast by moving `setNumDatanodes` off the chain.
   
   ```suggestion
       MiniOzoneHAClusterImpl.Builder builder = MiniOzoneCluster.newHABuilder(conf)
           .setSCMServiceId("scm-service-id")
           .setOMServiceId("om-service-id")
           .setNumOfActiveOMs(0)
           .setNumOfStorageContainerManagers(scmInstancesCount)
           .setNumOfActiveSCMs(1);
       builder.setNumDatanodes(0);
   
       // start single SCM instance without other Ozone services
       // in order to initialize and bootstrap SCM instances only
       cluster = builder.build();
   
       List<StorageContainerManager> storageContainerManagersList =
           cluster.getStorageContainerManagersList();
   ```



##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -323,4 +335,75 @@ private void waitForLeaderToBeReady()
     }, 1000, (int) ScmConfigKeys
         .OZONE_SCM_HA_RATIS_LEADER_READY_WAIT_TIMEOUT_DEFAULT);
   }
+
+  @Test
+  public void testSCMLeadershipMetric() throws IOException, InterruptedException {
+    // GIVEN
+    int scmInstancesCount = 3;
+    conf = new OzoneConfiguration();
+    MiniOzoneCluster.Builder haMiniClusterBuilder = MiniOzoneCluster.newHABuilder(conf)
+        .setSCMServiceId("scm-service-id")
+        .setOMServiceId("om-service-id")
+        .setNumOfActiveOMs(0)
+        .setNumOfStorageContainerManagers(scmInstancesCount)
+        .setNumOfActiveSCMs(1)
+        .setNumDatanodes(0);
+
+    // start single SCM instance without other Ozone services
+    // in order to initialize and bootstrap SCM instances only
+    cluster = (MiniOzoneHAClusterImpl) haMiniClusterBuilder.build();
+
+    List<StorageContainerManager> storageContainerManagersList =
+        ((MiniOzoneHAClusterImpl) cluster).getStorageContainerManagersList();
+
+    // stop the single SCM instance in order to imitate further simultaneous start of SCMs
+    storageContainerManagersList.get(0).stop();
+    storageContainerManagersList.get(0).join();
+
+    // WHEN (imitate simultaneous start of the SCMs)
+    CountDownLatch scmInstancesCounter = new CountDownLatch(scmInstancesCount);
+    int retryCount = 0;
+    while (true) {
+      AtomicInteger failedSCMs = new AtomicInteger();

Review Comment:
   `scmInstancesCounter` can be defined inside the loop, just like `failedSCMs`.  Then we don't need `finalScmInstancesCounter` and reassignment in `else`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "myskov (via GitHub)" <gi...@apache.org>.
myskov commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1853818342


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java:
##########
@@ -1595,8 +1595,22 @@ public void start() throws IOException {
 
     setStartTime();
 
-    // At this point leader is not known
-    scmHAMetricsUpdate(null);
+    AtomicBoolean leaderIsDefined = new AtomicBoolean(false);
+    if (SCMHAUtils.isSCMHAEnabled(configuration)) {
+      getScmHAManager().getRatisServer().getRatisRoles().stream()

Review Comment:
   There's the following method for OM: OzoneManagerRatisServer.getLeaderId
   maybe we should add the same for SCMRatisServer? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. Update metrics with leaderId if known when starting SCM [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7471:
URL: https://github.com/apache/ozone/pull/7471#issuecomment-2508030064

   Thanks @vtutrinov for iterating on the patch, @myskov for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1862993833


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -72,6 +77,8 @@
 @Timeout(300)
 public class TestStorageContainerManagerHA {
 
+  private static final Logger LOG = LoggerFactory.getLogger(TestMiniOzoneCluster.class);

Review Comment:
   Done



##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHA.java:
##########
@@ -323,4 +335,75 @@ private void waitForLeaderToBeReady()
     }, 1000, (int) ScmConfigKeys
         .OZONE_SCM_HA_RATIS_LEADER_READY_WAIT_TIMEOUT_DEFAULT);
   }
+
+  @Test
+  public void testSCMLeadershipMetric() throws IOException, InterruptedException {
+    // GIVEN
+    int scmInstancesCount = 3;
+    conf = new OzoneConfiguration();
+    MiniOzoneCluster.Builder haMiniClusterBuilder = MiniOzoneCluster.newHABuilder(conf)
+        .setSCMServiceId("scm-service-id")
+        .setOMServiceId("om-service-id")
+        .setNumOfActiveOMs(0)
+        .setNumOfStorageContainerManagers(scmInstancesCount)
+        .setNumOfActiveSCMs(1)
+        .setNumDatanodes(0);
+
+    // start single SCM instance without other Ozone services
+    // in order to initialize and bootstrap SCM instances only
+    cluster = (MiniOzoneHAClusterImpl) haMiniClusterBuilder.build();
+
+    List<StorageContainerManager> storageContainerManagersList =
+        ((MiniOzoneHAClusterImpl) cluster).getStorageContainerManagersList();

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1856076728


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java:
##########
@@ -584,19 +586,29 @@ protected SCMHAService createSCMService()
             }
             scmList.add(scm);
 
-            if (i <= numOfActiveSCMs) {
-              scm.start();
-              activeSCMs.add(scm);
-              LOG.info("Started SCM RPC server at {}",
-                  scm.getClientRpcAddress());
-            } else {
-              inactiveSCMs.add(scm);
-              LOG.info("Intialized SCM at {}. This SCM is currently "
-                  + "inactive (not running).", scm.getClientRpcAddress());
-            }
+            int scmCounter = i;
+            new Thread(() -> {
+              if (scmCounter <= numOfActiveSCMs) {
+                try {
+                  scm.start();
+                } catch (IOException e) {
+                  scmStartCounter.countDown();
+                  throw new RuntimeException(e);
+                }
+                activeSCMs.add(scm);
+                LOG.info("Started SCM RPC server at {}",
+                    scm.getClientRpcAddress());
+              } else {
+                inactiveSCMs.add(scm);
+                LOG.info("Initialized SCM at {}. This SCM is currently "
+                    + "inactive (not running).", scm.getClientRpcAddress());
+              }
+              scmStartCounter.countDown();
+            }).start();

Review Comment:
   Can you please explain why SCM start is moved to its own thread?



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMRatisServerImpl.java:
##########
@@ -147,6 +148,15 @@ public GrpcTlsConfig getGrpcTlsConfig() {
     return grpcTlsConfig;
   }
 
+  @Override
+  @Nullable
+  public RaftPeerId getLeaderId() {
+    if (getLeader() != null) {
+      return getLeader().getId();

Review Comment:
   `getLeader()` only once.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1856270115


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java:
##########
@@ -584,19 +586,29 @@ protected SCMHAService createSCMService()
             }
             scmList.add(scm);
 
-            if (i <= numOfActiveSCMs) {
-              scm.start();
-              activeSCMs.add(scm);
-              LOG.info("Started SCM RPC server at {}",
-                  scm.getClientRpcAddress());
-            } else {
-              inactiveSCMs.add(scm);
-              LOG.info("Intialized SCM at {}. This SCM is currently "
-                  + "inactive (not running).", scm.getClientRpcAddress());
-            }
+            int scmCounter = i;
+            new Thread(() -> {
+              if (scmCounter <= numOfActiveSCMs) {
+                try {
+                  scm.start();
+                } catch (IOException e) {
+                  scmStartCounter.countDown();
+                  throw new RuntimeException(e);
+                }
+                activeSCMs.add(scm);
+                LOG.info("Started SCM RPC server at {}",
+                    scm.getClientRpcAddress());
+              } else {
+                inactiveSCMs.add(scm);
+                LOG.info("Initialized SCM at {}. This SCM is currently "
+                    + "inactive (not running).", scm.getClientRpcAddress());
+              }
+              scmStartCounter.countDown();
+            }).start();

Review Comment:
   > Can we do that in a single test case, setting active SCMs to 0, and starting all 3 SCMs from that test in separate threads?
   
   Yep, sure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1861570782


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java:
##########
@@ -584,19 +586,29 @@ protected SCMHAService createSCMService()
             }
             scmList.add(scm);
 
-            if (i <= numOfActiveSCMs) {
-              scm.start();
-              activeSCMs.add(scm);
-              LOG.info("Started SCM RPC server at {}",
-                  scm.getClientRpcAddress());
-            } else {
-              inactiveSCMs.add(scm);
-              LOG.info("Intialized SCM at {}. This SCM is currently "
-                  + "inactive (not running).", scm.getClientRpcAddress());
-            }
+            int scmCounter = i;
+            new Thread(() -> {
+              if (scmCounter <= numOfActiveSCMs) {
+                try {
+                  scm.start();
+                } catch (IOException e) {
+                  scmStartCounter.countDown();
+                  throw new RuntimeException(e);
+                }
+                activeSCMs.add(scm);
+                LOG.info("Started SCM RPC server at {}",
+                    scm.getClientRpcAddress());
+              } else {
+                inactiveSCMs.add(scm);
+                LOG.info("Initialized SCM at {}. This SCM is currently "
+                    + "inactive (not running).", scm.getClientRpcAddress());
+              }
+              scmStartCounter.countDown();
+            }).start();

Review Comment:
   Done
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1861717824


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java:
##########


Review Comment:
   Done



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java:
##########
@@ -1595,8 +1595,12 @@ public void start() throws IOException {
 
     setStartTime();
 
-    // At this point leader is not known
-    scmHAMetricsUpdate(null);
+    if (SCMHAUtils.isSCMHAEnabled(configuration) && getScmHAManager().getRatisServer().getLeaderId() != null) {
+      scmHAMetricsUpdate(getScmHAManager().getRatisServer().getLeaderId().toString());
+    } else {
+      // At this point leader is not known
+      scmHAMetricsUpdate(null);
+    }

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1856216033


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java:
##########
@@ -584,19 +586,29 @@ protected SCMHAService createSCMService()
             }
             scmList.add(scm);
 
-            if (i <= numOfActiveSCMs) {
-              scm.start();
-              activeSCMs.add(scm);
-              LOG.info("Started SCM RPC server at {}",
-                  scm.getClientRpcAddress());
-            } else {
-              inactiveSCMs.add(scm);
-              LOG.info("Intialized SCM at {}. This SCM is currently "
-                  + "inactive (not running).", scm.getClientRpcAddress());
-            }
+            int scmCounter = i;
+            new Thread(() -> {
+              if (scmCounter <= numOfActiveSCMs) {
+                try {
+                  scm.start();
+                } catch (IOException e) {
+                  scmStartCounter.countDown();
+                  throw new RuntimeException(e);
+                }
+                activeSCMs.add(scm);
+                LOG.info("Started SCM RPC server at {}",
+                    scm.getClientRpcAddress());
+              } else {
+                inactiveSCMs.add(scm);
+                LOG.info("Initialized SCM at {}. This SCM is currently "
+                    + "inactive (not running).", scm.getClientRpcAddress());
+              }
+              scmStartCounter.countDown();
+            }).start();

Review Comment:
   To imitate the simultaneous start of the number of SCMs. In integration tests it was impossible to catch a case when the metrics were set to 0, but it's possible if the SCMs start simultaneously



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1856218646


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneHAClusterImpl.java:
##########
@@ -584,19 +586,29 @@ protected SCMHAService createSCMService()
             }
             scmList.add(scm);
 
-            if (i <= numOfActiveSCMs) {
-              scm.start();
-              activeSCMs.add(scm);
-              LOG.info("Started SCM RPC server at {}",
-                  scm.getClientRpcAddress());
-            } else {
-              inactiveSCMs.add(scm);
-              LOG.info("Intialized SCM at {}. This SCM is currently "
-                  + "inactive (not running).", scm.getClientRpcAddress());
-            }
+            int scmCounter = i;
+            new Thread(() -> {
+              if (scmCounter <= numOfActiveSCMs) {
+                try {
+                  scm.start();
+                } catch (IOException e) {
+                  scmStartCounter.countDown();
+                  throw new RuntimeException(e);
+                }
+                activeSCMs.add(scm);
+                LOG.info("Started SCM RPC server at {}",
+                    scm.getClientRpcAddress());
+              } else {
+                inactiveSCMs.add(scm);
+                LOG.info("Initialized SCM at {}. This SCM is currently "
+                    + "inactive (not running).", scm.getClientRpcAddress());
+              }
+              scmStartCounter.countDown();
+            }).start();

Review Comment:
   Can we do that in a single test case, setting active SCMs to 0, and starting all 3 SCMs from that test in separate threads?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11742. don't reset SCM leadership metric to null on initilizing http server [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7471:
URL: https://github.com/apache/ozone/pull/7471#discussion_r1861645323


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java:
##########


Review Comment:
   I think changes in this file are no longer needed, since the new test case uses 0 datanodes.
   
   We'll need to revisit datanode version mocking separately.  (HDDS-11819)



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java:
##########
@@ -1595,8 +1595,12 @@ public void start() throws IOException {
 
     setStartTime();
 
-    // At this point leader is not known
-    scmHAMetricsUpdate(null);
+    if (SCMHAUtils.isSCMHAEnabled(configuration) && getScmHAManager().getRatisServer().getLeaderId() != null) {
+      scmHAMetricsUpdate(getScmHAManager().getRatisServer().getLeaderId().toString());
+    } else {
+      // At this point leader is not known
+      scmHAMetricsUpdate(null);
+    }

Review Comment:
   Sorry, I didn't notice this earlier.  Please call `getLeaderId()` once, and use it in both the condition and to update metrics.
   
   ```suggestion
       RaftPeerId leaderId = SCMHAUtils.isSCMHAEnabled(configuration)
           ? getScmHAManager().getRatisServer().getLeaderId()
           : null;
       scmHAMetricsUpdate(Objects.toString(leaderId, null));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org