You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "GrantPSpencer (via GitHub)" <gi...@apache.org> on 2024/09/03 18:41:19 UTC

[PR] Add functionality to forcefully kill an instance [helix]

GrantPSpencer opened a new pull request, #2898:
URL: https://github.com/apache/helix/pull/2898

   ### Issues
   
   - [ ] My PR addresses the following Helix issues and references them in the PR description:
   
   N/A - New feature to forcefully kill an instance
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI changes:
   
   This PR adds a new feature, a HelixAdmin and Helix-rest API command to forcefully kill an instance. This is achieved by marking the instance's operation as UNKNOWN and then deleting the LIVEINSTANCE znode. This feature is intended for use in a scenario where the participant is in an unrecoverable state but is keeping an active connection with ZK. Marking the node as UNKNOWN will remove it from calculations and subsequently deleting the LIVEINSTANCE znode will cause the controller to consider it as OFFLINE. This skips the requirement that the node must process the downward state transition for topstate handoff to occur. 
   
   My current findings indicate that the LIVEINSTANCE znode will only be recreated on ZK session establishment, which occurs on initial connection and after session expiration. 
   
   #### The following code changes were made:
   
   * `helix-core/src/main/java/org/apache/helix/HelixAdmin.java`: Added `forceKillInstance` method to the HelixAdmin interface.
   * `helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java`: Implemented the `forceKillInstance` method in the ZKHelixAdmin class.
   * `helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/PerInstanceAccessor.java`: Added forceKillInstance command to the to the REST API updateInstance endpoint. Called via:
   ```
   https://<helix-rest-url>/namespaces/<namespace>/clusters/<cluster>/instances/<instance>?command=forceKillInstance
   ```
   
   Also includes miscellaneous changes:
   * `helix-core/src/test/java/org/apache/helix/integration/rebalancer/TestInstanceOperation.java` Corrected the logger class reference.
   * `helix-rest/src/test/java/org/apache/helix/rest/server/TestPartitionAssignmentAPI.java`: Corrected the logger class reference.
   * `helix-rest/src/test/java/org/apache/helix/rest/server/AbstractTestClass.java`: Refactored resource creation logic. Added addParticipant and dropParticipant methods. Also added another test cluster to isolate `testPerInstanceAccessor` and `testInstancesAccessor`
   * `helix-rest/src/test/java/org/apache/helix/rest/server/TestInstancesAccessor.java`: Now using isolated test cluster
   
   ### Tests
   
   - [ ] The following tests are written for this issue:
   
   * `helix-core/src/test/java/org/apache/helix/integration/TestForceKillInstance.java` for HelixAdmin API
   ```
   $ mvn test -o -Dtest=TestForceKillInstance -pl=helix-core
   
   [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.027 s - in org.apache.helix.integration.TestForceKillInstance
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  01:16 min
   [INFO] Finished at: 2024-09-03T11:37:35-07:00
   [INFO] ------------------------------------------------------------------------
   
   ```
   
   * `testForceKillInstance` in `helix-rest/src/test/java/org/apache/helix/rest/server/TestPerInstanceAccessor.java` for Helix-Rest API
   ```
   $ mvn test -o -Dtest=TestInstancesAccessor -pl=helix-rest  
   
   [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.395 s - in org.apache.helix.rest.server.TestInstancesAccessor
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  01:40 min
   [INFO] Finished at: 2024-09-03T11:35:04-07:00
   [INFO] ------------------------------------------------------------------------
   ```
   
   ### Changes that Break Backward Compatibility (Optional)
   
   - My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
   N/A
   
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on PR #2898:
URL: https://github.com/apache/helix/pull/2898#issuecomment-2332222768

   Pull request approved by: @junkaixue 
   Commit message: Add functionality to forcefully kill an instance


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2898:
URL: https://github.com/apache/helix/pull/2898#discussion_r1744372615


##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java:
##########
@@ -731,6 +731,36 @@ public boolean isReadyForPreparingJoiningCluster(String clusterName, String inst
     return false;
   }
 
+  @Override
+  public boolean forceKillInstance(String clusterName, String instanceName) {
+    return forceKillInstance(clusterName, instanceName, "Force kill instance", null);

Review Comment:
   forceKillInstance method allows users to specify the reason and operationSource of the UNKNOWN instance operation that is set as part of the forceKillInstance logic. If no reason is passed, I figured it would still be a good idea to clarify that the UNKNOWN operation was set by the forceKillInstance path



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on PR #2898:
URL: https://github.com/apache/helix/pull/2898#issuecomment-2329920532

   Currently triggering manual runs on my fork's CI to confirm no flaky tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2898:
URL: https://github.com/apache/helix/pull/2898#discussion_r1744375247


##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java:
##########
@@ -731,6 +731,36 @@ public boolean isReadyForPreparingJoiningCluster(String clusterName, String inst
     return false;
   }
 
+  @Override
+  public boolean forceKillInstance(String clusterName, String instanceName) {
+    return forceKillInstance(clusterName, instanceName, "Force kill instance", null);
+  }
+
+  @Override
+  public boolean forceKillInstance(String clusterName, String instanceName, String reason,
+      InstanceConstants.InstanceOperationSource operationSource) {
+    logger.info("Force kill instance {} in cluster {}.", instanceName, clusterName);
+    HelixDataAccessor accessor = new ZKHelixDataAccessor(clusterName, _baseDataAccessor);
+
+    // TODO: Can we batch Instance Operation change and LIVEINSTANCES znode deletion?

Review Comment:
   Switched to using _zkClient.multi(). 
   Tried using baseDataAccessor.multiSet(), but could not figure out how to get DataUpdater to delete the LIVEINSTANCE ZNode



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on code in PR #2898:
URL: https://github.com/apache/helix/pull/2898#discussion_r1742592191


##########
helix-core/src/main/java/org/apache/helix/HelixAdmin.java:
##########
@@ -844,4 +844,20 @@ default boolean isReadyForPreparingJoiningCluster(String clusterName, String ins
     throw new UnsupportedOperationException(
         "isReadyForPreparingJoiningCluster is not implemented.");
   }
+
+  /**
+   * Attempt to forcefully kill an instance in the cluster. This will remove the instance's

Review Comment:
   Put a warning here. Besides the session timeout, users need to handle the instance without live node properly.



##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java:
##########
@@ -731,6 +731,36 @@ public boolean isReadyForPreparingJoiningCluster(String clusterName, String inst
     return false;
   }
 
+  @Override
+  public boolean forceKillInstance(String clusterName, String instanceName) {
+    return forceKillInstance(clusterName, instanceName, "Force kill instance", null);
+  }
+
+  @Override
+  public boolean forceKillInstance(String clusterName, String instanceName, String reason,
+      InstanceConstants.InstanceOperationSource operationSource) {
+    logger.info("Force kill instance {} in cluster {}.", instanceName, clusterName);
+    HelixDataAccessor accessor = new ZKHelixDataAccessor(clusterName, _baseDataAccessor);
+
+    // TODO: Can we batch Instance Operation change and LIVEINSTANCES znode deletion?

Review Comment:
   Make the solution more clear: use multi set to make sure the update operation and delete node in the same operation.



##########
helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixAdmin.java:
##########
@@ -731,6 +731,36 @@ public boolean isReadyForPreparingJoiningCluster(String clusterName, String inst
     return false;
   }
 
+  @Override
+  public boolean forceKillInstance(String clusterName, String instanceName) {
+    return forceKillInstance(clusterName, instanceName, "Force kill instance", null);

Review Comment:
   For kill by default as the message?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on PR #2898:
URL: https://github.com/apache/helix/pull/2898#issuecomment-2332220109

   @GrantPSpencer ready to checkin?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "GrantPSpencer (via GitHub)" <gi...@apache.org>.
GrantPSpencer commented on code in PR #2898:
URL: https://github.com/apache/helix/pull/2898#discussion_r1744368708


##########
helix-core/src/main/java/org/apache/helix/HelixAdmin.java:
##########
@@ -844,4 +844,20 @@ default boolean isReadyForPreparingJoiningCluster(String clusterName, String ins
     throw new UnsupportedOperationException(
         "isReadyForPreparingJoiningCluster is not implemented.");
   }
+
+  /**
+   * Attempt to forcefully kill an instance in the cluster. This will remove the instance's

Review Comment:
   Added warning to javadoc
   ```
      * WARNING: THIS METHOD WILL FORCEFULLY DROP THE NODE FROM THE EXTERNAL VIEW WITHOUT WAITING FOR IT TO PROCESS
      * ANY DOWNWARD STATE TRANSITIONS. USE WITH CAUTION AS THE NODE WILL KEEP ITS CURRENT STATES LOCALLY AND NOT DOWNWARD
      * ST UNTIL IT RECONNECTS UNDER A NEW SESSION AND RESETS TO THE INITIALSTATE.
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Add functionality to forcefully kill an instance [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue merged PR #2898:
URL: https://github.com/apache/helix/pull/2898


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org