You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/06/20 13:22:04 UTC

[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1102: HDDS-3825. Show rule status of safemode

maobaolong opened a new pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102


   ## What changes were proposed in this pull request?
   
   Without this PR, we can't get the reason why scm still in the safemode, now, we can get the each rule status, and know the condition to exit safemode.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-3825
   
   ## How was this patch tested?
   
   ```bash
   ozone admin safemode status --verbose
   SCM is in safe mode.
   validated:false, DataNodeSafeModeRule, registeredDns 0 should >= requiredDns 1
   validated:true, HealthyPipelineSafeModeRule, currentHealthyPipelineCount 0 should >= healthyPipelineThresholdCount 0
   validated:true, ContainerSafeModeRule, currentContainerThreshold 1.0 should >= safeModeCutoff 0.99
   validated:true, AtleastOneDatanodeReportedRule, currentReportedPipelineCount 0 should >= thresholdCount 0
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1102: HDDS-3825. Show rule status of SCM safemode

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#issuecomment-647838114


   @dineshchitlangia Thanks you for create the ticket for me, i'd love to do it with pleasure.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1102: HDDS-3825. Show rule status of SCM safemode

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#issuecomment-647275381


   # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1102?src=pr&el=h1) Report
   > Merging [#1102](https://codecov.io/gh/apache/hadoop-ozone/pull/1102?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hadoop-ozone/commit/8b2f9f41eeec9ed29ebc45adfe11a713f29bdc28&el=desc) will **decrease** coverage by `0.16%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/graphs/tree.svg?width=650&height=150&src=pr&token=5YeeptJMby)](https://codecov.io/gh/apache/hadoop-ozone/pull/1102?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1102      +/-   ##
   ============================================
   - Coverage     70.61%   70.45%   -0.17%     
   + Complexity     9423     9406      -17     
   ============================================
     Files           965      965              
     Lines         49005    49050      +45     
     Branches       4797     4802       +5     
   ============================================
   - Hits          34604    34557      -47     
   - Misses        12097    12177      +80     
   - Partials       2304     2316      +12     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/hadoop-ozone/pull/1102?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...ntainerLocationProtocolClientSideTranslatorPB.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvZnJhbWV3b3JrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9zY20vcHJvdG9jb2xQQi9TdG9yYWdlQ29udGFpbmVyTG9jYXRpb25Qcm90b2NvbENsaWVudFNpZGVUcmFuc2xhdG9yUEIuamF2YQ==) | `42.79% <0.00%> (-2.31%)` | `22.00 <0.00> (ø)` | |
   | [...ntainerLocationProtocolServerSideTranslatorPB.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3Byb3RvY29sL1N0b3JhZ2VDb250YWluZXJMb2NhdGlvblByb3RvY29sU2VydmVyU2lkZVRyYW5zbGF0b3JQQi5qYXZh) | `40.61% <0.00%> (-2.85%)` | `19.00 <0.00> (ø)` | |
   | [...adoop/hdds/scm/safemode/ContainerSafeModeRule.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NhZmVtb2RlL0NvbnRhaW5lclNhZmVNb2RlUnVsZS5qYXZh) | `82.35% <ø> (-2.50%)` | `15.00 <0.00> (ø)` | |
   | [...hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NhZmVtb2RlL0RhdGFOb2RlU2FmZU1vZGVSdWxlLmphdmE=) | `87.50% <ø> (-5.84%)` | `6.00 <0.00> (ø)` | |
   | [...hdds/scm/safemode/HealthyPipelineSafeModeRule.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NhZmVtb2RlL0hlYWx0aHlQaXBlbGluZVNhZmVNb2RlUnVsZS5qYXZh) | `88.09% <ø> (-2.15%)` | `13.00 <0.00> (ø)` | |
   | [...s/scm/safemode/OneReplicaPipelineSafeModeRule.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NhZmVtb2RlL09uZVJlcGxpY2FQaXBlbGluZVNhZmVNb2RlUnVsZS5qYXZh) | `84.61% <ø> (-2.23%)` | `12.00 <0.00> (ø)` | |
   | [...e/hadoop/hdds/scm/safemode/SCMSafeModeManager.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NhZmVtb2RlL1NDTVNhZmVNb2RlTWFuYWdlci5qYXZh) | `88.11% <0.00%> (-5.57%)` | `26.00 <0.00> (ø)` | |
   | [...che/hadoop/hdds/scm/safemode/SafeModeExitRule.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NhZmVtb2RlL1NhZmVNb2RlRXhpdFJ1bGUuamF2YQ==) | `94.44% <ø> (ø)` | `7.00 <0.00> (ø)` | |
   | [...adoop/hdds/scm/server/SCMClientProtocolServer.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NlcnZlci9TQ01DbGllbnRQcm90b2NvbFNlcnZlci5qYXZh) | `50.97% <0.00%> (-0.20%)` | `27.00 <0.00> (ø)` | |
   | [...adoop/hdds/scm/server/StorageContainerManager.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL3NlcnZlci9TdG9yYWdlQ29udGFpbmVyTWFuYWdlci5qYXZh) | `75.00% <0.00%> (-0.21%)` | `66.00 <0.00> (ø)` | |
   | ... and [25 more](https://codecov.io/gh/apache/hadoop-ozone/pull/1102/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1102?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1102?src=pr&el=footer). Last update [8b2f9f4...0bbdd05](https://codecov.io/gh/apache/hadoop-ozone/pull/1102?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #1102: HDDS-3825. Show rule status of SCM safemode

Posted by GitBox <gi...@apache.org>.
dineshchitlangia merged pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1102: HDDS-3825. Show rule status of SCM safemode

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#issuecomment-647256339


   @dineshchitlangia Thanks for your +1. 😃


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #1102: HDDS-3825. Show rule status of safemode

Posted by GitBox <gi...@apache.org>.
dineshchitlangia commented on a change in pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#discussion_r443294463



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/ContainerSafeModeRule.java
##########
@@ -125,4 +125,10 @@ protected void process(NodeRegistrationContainerReport reportsProto) {
   protected void cleanup() {
     containerMap.clear();
   }
+
+  @Override
+  public String getStatusText() {
+    return "currentContainerThreshold " + getCurrentContainerThreshold()
+        + " should >= safeModeCutoff " + this.safeModeCutoff;

Review comment:
       ```suggestion
           + " >= safeModeCutoff " + this.safeModeCutoff;
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java
##########
@@ -79,4 +79,10 @@ protected void process(NodeRegistrationContainerReport reportsProto) {
   protected void cleanup() {
     registeredDnSet.clear();
   }
+
+  @Override
+  public String getStatusText() {
+    return "registeredDns " + this.registeredDns + " should >= requiredDns "

Review comment:
       ```suggestion
       return "registeredDns " + this.registeredDns + " >= requiredDns "
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SCMSafeModeManager.java
##########
@@ -272,6 +273,21 @@ public boolean getInSafeMode() {
     return inSafeMode.get();
   }
 
+  /**
+   * Get the safe mode statuses of each rules.

Review comment:
       ```suggestion
      * Get the safe mode status of all rules.
   ```

##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
##########
@@ -210,6 +212,15 @@ Pipeline createReplicationPipeline(HddsProtos.ReplicationType type,
    */
   boolean inSafeMode() throws IOException;
 
+  /**
+   * Get the safe mode statuses of each rules.

Review comment:
       ```suggestion
      * Get the safe mode status of all rules.
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/ContainerSafeModeRule.java
##########
@@ -125,4 +125,10 @@ protected void process(NodeRegistrationContainerReport reportsProto) {
   protected void cleanup() {
     containerMap.clear();
   }
+
+  @Override
+  public String getStatusText() {
+    return "currentContainerThreshold " + getCurrentContainerThreshold()
+        + " should >= safeModeCutoff " + this.safeModeCutoff;
+  }
 }

Review comment:
       ```suggestion
   }
   
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/DataNodeSafeModeRule.java
##########
@@ -79,4 +79,10 @@ protected void process(NodeRegistrationContainerReport reportsProto) {
   protected void cleanup() {
     registeredDnSet.clear();
   }
+
+  @Override
+  public String getStatusText() {
+    return "registeredDns " + this.registeredDns + " should >= requiredDns "
+        + this.requiredDns;
+  }
 }

Review comment:
       ```suggestion
   }
   
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/HealthyPipelineSafeModeRule.java
##########
@@ -150,4 +150,11 @@ public int getCurrentHealthyPipelineCount() {
   public int getHealthyPipelineThresholdCount() {
     return healthyPipelineThresholdCount;
   }
+
+  @Override
+  public String getStatusText() {
+    return "currentHealthyPipelineCount " + this.currentHealthyPipelineCount
+        + " should >= healthyPipelineThresholdCount "

Review comment:
       ```suggestion
           + " >= healthyPipelineThresholdCount "
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -1110,4 +1111,13 @@ public SCMMetadataStore getScmMetadataStore() {
   public NetworkTopology getClusterMap() {
     return this.clusterMap;
   }
+
+  /**
+   * Get the safe mode statuses of each rules.

Review comment:
       ```suggestion
      * Get the safe mode status of all rules.
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/HealthyPipelineSafeModeRule.java
##########
@@ -150,4 +150,11 @@ public int getCurrentHealthyPipelineCount() {
   public int getHealthyPipelineThresholdCount() {
     return healthyPipelineThresholdCount;
   }
+
+  @Override
+  public String getStatusText() {
+    return "currentHealthyPipelineCount " + this.currentHealthyPipelineCount
+        + " should >= healthyPipelineThresholdCount "
+        + this.healthyPipelineThresholdCount;
+  }
 }

Review comment:
       ```suggestion
   }
   
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/OneReplicaPipelineSafeModeRule.java
##########
@@ -134,4 +134,11 @@ public int getThresholdCount() {
   public int getCurrentReportedPipelineCount() {
     return currentReportedPipelineCount;
   }
+
+  @Override
+  public String getStatusText() {
+    return "currentReportedPipelineCount "
+        + this.currentReportedPipelineCount + " should >= thresholdCount "
+        + this.thresholdCount;
+  }
 }

Review comment:
       ```suggestion
   }
   
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SCMSafeModeManager.java
##########
@@ -272,6 +273,21 @@ public boolean getInSafeMode() {
     return inSafeMode.get();
   }
 
+  /**
+   * Get the safe mode statuses of each rules.
+   *
+   * @return map of rule statuses.
+   */
+  public Map<String, Pair<Boolean, String>> getRuleStatus() {
+    Map<String, Pair<Boolean, String>> map =
+        new HashMap<>();

Review comment:
       ```suggestion
       Map<String, Pair<Boolean, String>> map = new HashMap<>();
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModeExitRule.java
##########
@@ -111,4 +111,9 @@ protected SafeModeMetrics getSafeModeMetrics() {
     return safeModeManager.getSafeModeMetrics();
   }
 
+  /**
+   * Return's the status text of this rule.

Review comment:
       ```suggestion
      * Returns the status text of the rule.
   ```

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/OneReplicaPipelineSafeModeRule.java
##########
@@ -134,4 +134,11 @@ public int getThresholdCount() {
   public int getCurrentReportedPipelineCount() {
     return currentReportedPipelineCount;
   }
+
+  @Override
+  public String getStatusText() {
+    return "currentReportedPipelineCount "
+        + this.currentReportedPipelineCount + " should >= thresholdCount "

Review comment:
       ```suggestion
           + this.currentReportedPipelineCount + " >= thresholdCount "
   ```

##########
File path: hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeCheckSubcommand.java
##########
@@ -43,6 +46,10 @@
   @ParentCommand
   private SafeModeCommands parent;
 
+  @CommandLine.Option(names = {"--verbose"},
+      description = "More verbose output. Show the status of rules.")

Review comment:
       ```suggestion
         description = "Show detailed status of rules.")
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on pull request #1102: HDDS-3825. Show rule status of SCM safemode

Posted by GitBox <gi...@apache.org>.
dineshchitlangia commented on pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#issuecomment-647832161


   @maobaolong I filed https://issues.apache.org/jira/browse/HDDS-3849 and assigned to you Thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1102: HDDS-3825. Show rule status of safemode

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#issuecomment-647069455


   @elek Please take a look, thank you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1102: HDDS-3825. Show rule status of SCM safemode

Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#issuecomment-647247441


   @dineshchitlangia Thanks for your review and the suggestion, i've commit all of your suggestion commit, thank for your help. PTAL.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] dineshchitlangia commented on pull request #1102: HDDS-3825. Show rule status of SCM safemode

Posted by GitBox <gi...@apache.org>.
dineshchitlangia commented on pull request #1102:
URL: https://github.com/apache/hadoop-ozone/pull/1102#issuecomment-647830639


   @maobaolong I will merge this. However, going forward could you pls also include any relevant tests with your changes as the code coverage is now being looked at more seriously in the community for the benefits it brings :)
   For this patch, pls file another jira to add the tests.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org