You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "himanshukandwal (via GitHub)" <gi...@apache.org> on 2023/08/22 00:31:33 UTC

[GitHub] [helix] himanshukandwal opened a new pull request, #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

himanshukandwal opened a new pull request, #2603:
URL: https://github.com/apache/helix/pull/2603

   ### Issues
   
   - [X] My PR addresses the following Helix issues and references them in the PR description:
   #2590
   
   ### Description
   - [X] Here are some details about my PR, including screenshots of any UI changes:
   We encountered that with resetting of the the `missingTopStatePartitionsBeyondThresholdGauge` to 0 have created several problems:
   * Missed Events to Ingraphs: If the value of `missingTopStatePartitionsBeyondThresholdGauge` gets reset to 0 before the refresh rate of Ingraph metric collector.
   * Negative Value in Metrics: If the top-state instance bounces back after the reset, the decrement will bring the value to Negative. We have also added a check to ensure that no negative value ever gets set.
   
   ### Tests
   - [X] The following tests are written for this issue:
   * Updated TestTopStateHandoffMetrics.testTopStateFailedHandoff
   * Added TestTopStateHandoffMetrics.testTopStateFailedUnrecoveredHandoff
   
   - The following is the result of the "mvn test" command on the appropriate module:
   ```console
   [INFO] Results:
   [INFO] 
   [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
   [INFO] 
   [INFO] 
   [INFO] --- jacoco:0.8.6:report (generate-code-coverage-report) @ helix-core ---
   [INFO] Loading execution data file /Users/hkandwal/Documents/workspaces/projects/helix_os_hk/helix-core/target/jacoco.exec
   [INFO] Analyzed bundle 'Apache Helix :: Core' with 799 classes
   [WARNING] Classes in bundle 'Apache Helix :: Core' do not match with execution data. For report generation the same class files must be used as at runtime.
   [WARNING] Execution data for class org/apache/helix/controller/rebalancer/waged/WagedRebalancer does not match.
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  27.585 s
   [INFO] Finished at: 2023-08-21T17:30:55-07:00
   [INFO] ------------------------------------------------------------------------
   ```
   
   ### Changes that Break Backward Compatibility (Optional)
   
   - My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
   
   (Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
   
   ### Documentation (Optional)
   
   - In case of new functionality, my PR adds documentation in the following wiki page:
   
   (Link the GitHub wiki you added)
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on PR #2603:
URL: https://github.com/apache/helix/pull/2603#issuecomment-1690022334

   Please create another PR to track the Periodic pipeline related change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] himanshukandwal commented on a diff in pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2603:
URL: https://github.com/apache/helix/pull/2603#discussion_r1302532317


##########
helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ResourceMonitor.java:
##########
@@ -480,7 +480,6 @@ public long getRebalanceThrottledByErrorPartitionGauge() {
   public void resetMaxTopStateHandoffGauge() {
     if (_lastResetTime + DEFAULT_RESET_INTERVAL_MS <= System.currentTimeMillis()) {
       _maxSinglePartitionTopStateHandoffDuration.updateValue(0L);
-      _missingTopStatePartitionsBeyondThresholdGauge.updateValue(0L);

Review Comment:
   Hey Rahul, Thanks for your review.  
   
   Yes as this fix is working fine and is urgent for WAGED rollout blocker, I will address this concern in a following PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] himanshukandwal commented on a diff in pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2603:
URL: https://github.com/apache/helix/pull/2603#discussion_r1303306765


##########
helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ResourceMonitor.java:
##########
@@ -480,7 +480,6 @@ public long getRebalanceThrottledByErrorPartitionGauge() {
   public void resetMaxTopStateHandoffGauge() {
     if (_lastResetTime + DEFAULT_RESET_INTERVAL_MS <= System.currentTimeMillis()) {
       _maxSinglePartitionTopStateHandoffDuration.updateValue(0L);
-      _missingTopStatePartitionsBeyondThresholdGauge.updateValue(0L);

Review Comment:
   done @desaikomal, thanks for pointing it out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] himanshukandwal commented on a diff in pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2603:
URL: https://github.com/apache/helix/pull/2603#discussion_r1302532317


##########
helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ResourceMonitor.java:
##########
@@ -480,7 +480,6 @@ public long getRebalanceThrottledByErrorPartitionGauge() {
   public void resetMaxTopStateHandoffGauge() {
     if (_lastResetTime + DEFAULT_RESET_INTERVAL_MS <= System.currentTimeMillis()) {
       _maxSinglePartitionTopStateHandoffDuration.updateValue(0L);
-      _missingTopStatePartitionsBeyondThresholdGauge.updateValue(0L);

Review Comment:
   Hey Rahul, Thanks for your review.  
   
   Yes as this fix is working fine and is urgent for Espresso WAGED rollout blocker, I will address this concern in a following PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] himanshukandwal commented on pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on PR #2603:
URL: https://github.com/apache/helix/pull/2603#issuecomment-1690320534

   This PR is reviewed and approved by @desaikomal .
   Commit message: Fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] himanshukandwal commented on a diff in pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "himanshukandwal (via GitHub)" <gi...@apache.org>.
himanshukandwal commented on code in PR #2603:
URL: https://github.com/apache/helix/pull/2603#discussion_r1302269166


##########
helix-core/src/main/java/org/apache/helix/controller/stages/TopStateHandoffReportStage.java:
##########
@@ -494,6 +494,9 @@ private void reportTopStateComesBack(ResourceControllerDataProvider cache, Map<S
 
     // In case of recovery after failure, we should decrement the missingTopStateBeyondThresholdGauge value.
     if (clusterStatusMonitor != null && record.isFailed()) {
+      LogUtil.logInfo(LOG, _eventId, String.format(
+          "Missing top state recovered for resource %s and partition %s. Decrementing missingTopStateBeyondThresholdGauge.",
+          resourceName, partition.getPartitionName()));

Review Comment:
   This wont be published all the time, and will be published only when the threshold is missed (and the record is set to failed).  
   We are publishing similar log during increment as well, to ensure that metric is emitted, given this is a very critical metric for end-users.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu merged pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2603:
URL: https://github.com/apache/helix/pull/2603


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on a diff in pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on code in PR #2603:
URL: https://github.com/apache/helix/pull/2603#discussion_r1303070878


##########
helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ResourceMonitor.java:
##########
@@ -480,7 +480,6 @@ public long getRebalanceThrottledByErrorPartitionGauge() {
   public void resetMaxTopStateHandoffGauge() {
     if (_lastResetTime + DEFAULT_RESET_INTERVAL_MS <= System.currentTimeMillis()) {
       _maxSinglePartitionTopStateHandoffDuration.updateValue(0L);
-      _missingTopStatePartitionsBeyondThresholdGauge.updateValue(0L);

Review Comment:
   please remove customer name from above. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] rahulrane50 commented on a diff in pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "rahulrane50 (via GitHub)" <gi...@apache.org>.
rahulrane50 commented on code in PR #2603:
URL: https://github.com/apache/helix/pull/2603#discussion_r1302299990


##########
helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ResourceMonitor.java:
##########
@@ -480,7 +480,6 @@ public long getRebalanceThrottledByErrorPartitionGauge() {
   public void resetMaxTopStateHandoffGauge() {
     if (_lastResetTime + DEFAULT_RESET_INTERVAL_MS <= System.currentTimeMillis()) {
       _maxSinglePartitionTopStateHandoffDuration.updateValue(0L);
-      _missingTopStatePartitionsBeyondThresholdGauge.updateValue(0L);

Review Comment:
   Had a offline discussion. Let's check about what happens if cache gets dropped as a part of another pipeline (periodic rebalance etc) before we remove this reset part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a diff in pull request #2603: [apache/helix] -- fixes #2590 Removed resetting of the missingTopStatePartitionsBeyondThresholdGauge to avoid incorrect accounting.

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2603:
URL: https://github.com/apache/helix/pull/2603#discussion_r1302236256


##########
helix-core/src/main/java/org/apache/helix/controller/stages/TopStateHandoffReportStage.java:
##########
@@ -494,6 +494,9 @@ private void reportTopStateComesBack(ResourceControllerDataProvider cache, Map<S
 
     // In case of recovery after failure, we should decrement the missingTopStateBeyondThresholdGauge value.
     if (clusterStatusMonitor != null && record.isFailed()) {
+      LogUtil.logInfo(LOG, _eventId, String.format(
+          "Missing top state recovered for resource %s and partition %s. Decrementing missingTopStateBeyondThresholdGauge.",
+          resourceName, partition.getPartitionName()));

Review Comment:
   Top state comes back is pretty frequent.  Will we flood log?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org