You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "ChenSammi (via GitHub)" <gi...@apache.org> on 2024/04/30 07:59:42 UTC

[PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

ChenSammi opened a new pull request, #1072:
URL: https://github.com/apache/ratis/pull/1072

   ## What changes were proposed in this pull request?
   Currently there is no way for application to get notified once the RaftServerProxy is close.
   
   This task is to add a close callback support to RaftServerProxy. 
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/RATIS-2066
   
   ## How was this patch tested?
   integration test with Ozone. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #1072:
URL: https://github.com/apache/ratis/pull/1072#issuecomment-2085840525

   @ChenSammi  , thanks for working on this!  
   
   We already have a `notifyServerShutdown`.  It will notify the `StateMachine` when its division (i.e. `RaftServerImpl`, not `RaftServerProxy`) is closed.  Is it good enough?
   https://github.com/apache/ratis/blob/1fe59e07dff9b02e46d150de8b46ed2d4fc3abce/ratis-server-api/src/main/java/org/apache/ratis/statemachine/StateMachine.java#L268
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #1072:
URL: https://github.com/apache/ratis/pull/1072


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #1072:
URL: https://github.com/apache/ratis/pull/1072#issuecomment-2093308221

   @ChenSammi , we may add a boolean parameter as below.  Anyway, we should use the same mechanism to notify the `StateMachine`.
   ```java
       /**
        * Notify the {@link StateMachine} that a raft server has step down.
        */
       default void notifyServerShutdown(RoleInfoProto roleInfo) {
         notifyServerShutdown(roleInfo, false);
       }
   
       default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers) {}
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #1072:
URL: https://github.com/apache/ratis/pull/1072#issuecomment-2096290715

   @ChenSammi , this JIRA is suggesting to add a callback.   Instead of adding a new callback, why not changing the `notifyServerShutdown` API and then call it in `ImplMap.close(groupId, future)` as below?
   ```java
   +++ b/ratis-server/src/main/java/org/apache/ratis/server/impl/RaftServerProxy.java
   @@ -138,6 +138,8 @@ class RaftServerProxy implements RaftServer {
          } catch (Throwable t) {
            LOG.warn("{}: Failed to close the division for {}", getId(), groupId, t);
          }
   +
   +      impl.getStateMachine().event().notifyServerShutdown(impl.getRoleInfoProto(), true);
        }
    
        synchronized List<RaftGroupId> getGroupIds() {
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #1072:
URL: https://github.com/apache/ratis/pull/1072#issuecomment-2114007865

   Thanks @szetszwo . 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #1072:
URL: https://github.com/apache/ratis/pull/1072#issuecomment-2095142790

   @szetszwo , notifyServerShutdown is called currently when a peer's in vote phase, and is found not in the raft group by leader, so it turns down itself.  
   
   The case we met in Ozone datanode is different.  The trace is,
     RaftServerProxy#pauseMonitor#handleJvmPause -> RaftServerProxy#close -> ImplMap for each Impl, RaftServerImpl#close
   
   None of StateMachine#EventApi is called now.  Are you suggesting to call notifyServerShutdown in RaftServerImpl#close, and give a reason why RaftServerImpl shutdown itself? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #1072:
URL: https://github.com/apache/ratis/pull/1072#issuecomment-2109556803

   @szetszwo ,  could you re-trigger the CI?  The failed TestSegmentedRaftLog passed locally. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #1072:
URL: https://github.com/apache/ratis/pull/1072#discussion_r1600428582


##########
ratis-server-api/src/main/java/org/apache/ratis/statemachine/StateMachine.java:
##########
@@ -266,6 +266,13 @@ default void notifySnapshotInstalled(InstallSnapshotResult result, long snapshot
      * Notify the {@link StateMachine} that a raft server has step down.
      */
     default void notifyServerShutdown(RoleInfoProto roleInfo) {}

Review Comment:
   We should deprecate the old method and change its default to
   ```java
       /**
        * The same as notifyServerShutdown(roleInfo, false).
        *
        * @deprecated Implementations should override {@link #notifyServerShutdown(RoleInfoProto, boolean)} instead.
        */
       @Deprecated
       default void notifyServerShutdown(RoleInfoProto roleInfo) {
         notifyServerShutdown(roleInfo, false);
       }
   ```
   Could you also update the caller?
   ```java
   +++ b/ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderElection.java
   @@ -334,7 +334,7 @@ class LeaderElection implements Runnable {
            case NOT_IN_CONF:
            case SHUTDOWN:
              server.close();
   -          server.getStateMachine().event().notifyServerShutdown(server.getRoleInfoProto());
   +          server.getStateMachine().event().notifyServerShutdown(server.getRoleInfoProto(), false);
              return false;
            case TIMEOUT:
   ```



##########
ratis-server-api/src/main/java/org/apache/ratis/statemachine/StateMachine.java:
##########
@@ -266,6 +266,13 @@ default void notifySnapshotInstalled(InstallSnapshotResult result, long snapshot
      * Notify the {@link StateMachine} that a raft server has step down.
      */
     default void notifyServerShutdown(RoleInfoProto roleInfo) {}
+
+    /**
+     * Notify the {@link StateMachine} that a raft server has been shutdown down.

Review Comment:
   `a raft server` seems inaccurate.  How about the following ?
   ```java
        * Notify the {@link StateMachine} that either the server for this division or all the servers have been shut down.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #1072:
URL: https://github.com/apache/ratis/pull/1072#discussion_r1601120991


##########
ratis-server-api/src/main/java/org/apache/ratis/statemachine/StateMachine.java:
##########
@@ -266,6 +266,13 @@ default void notifySnapshotInstalled(InstallSnapshotResult result, long snapshot
      * Notify the {@link StateMachine} that a raft server has step down.
      */
     default void notifyServerShutdown(RoleInfoProto roleInfo) {}

Review Comment:
   Sure. Addressed in new commit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RATIS-2066. RaftServerProxy supports close callback [ratis]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #1072:
URL: https://github.com/apache/ratis/pull/1072#issuecomment-2093098293

   @szetszwo , notifyServerShutdown is not enough for this DN case. For DN, there is always new pipeline created and old pipeline destroyed.  I guess this notifyServerShutdown will be called when an old pipeline is closed.  This specific DN case, is when there is long GC, longer than configured threshold, RaftServerProxy will close itself. When this happens, we need to close DN as well. To achieve this, one solution is provide an close callback as this JIRA do, another solution is we start a new thread in DN to closely monitor the RaftServerProxy state, whether it's running or closed.  After some thoughts, I prefer the first solution since it will save a thread resource. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org