You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by GitBox <gi...@apache.org> on 2022/05/09 14:38:59 UTC

[GitHub] [ratis] adoroszlai opened a new pull request, #643: RATIS-1577. Install snapshot failure

adoroszlai opened a new pull request, #643:
URL: https://github.com/apache/ratis/pull/643

   ## What changes were proposed in this pull request?
   
   After 7167fafe Ozone SCM HA fails to start due to the following error in follower:
   
   ```
   2022-05-09 15:06:39,907 [grpc-default-executor-0] INFO  impl.SnapshotInstallationHandler (SnapshotInstallationHandler.java:installSnapshot(79)) - 2d9f16d2-1d71-4978-9546-00aa402e881d@group-F84A6C219907: receive installSnapshot: edb17d6c-0f2e-4b73-aa2b-eb8fdf376958->2d9f16d2-1d71-4978-9546-00aa402e881d#0-t2,notify:(t:1, i:0)
   2022-05-09 15:06:39,909 [grpc-default-executor-0] INFO  server.RaftServer$Division (ServerState.java:setLeader(287)) - 2d9f16d2-1d71-4978-9546-00aa402e881d@group-F84A6C219907: change Leader from null to edb17d6c-0f2e-4b73-aa2b-eb8fdf376958 at term 2 for installSnapshot, leader elected after 488ms
   2022-05-09 15:06:39,910 [grpc-default-executor-0] INFO  impl.SnapshotInstallationHandler (SnapshotInstallationHandler.java:notifyStateMachineToInstallSnapshot(210)) - 2d9f16d2-1d71-4978-9546-00aa402e881d@group-F84A6C219907: Received notification to install snapshot at index 0
   2022-05-09 15:06:39,914 [grpc-default-executor-0] INFO  impl.SnapshotInstallationHandler (SnapshotInstallationHandler.java:notifyStateMachineToInstallSnapshot(243)) - 2d9f16d2-1d71-4978-9546-00aa402e881d@group-F84A6C219907: notifyInstallSnapshot: nextIndex is 0 but the leader's first available index is 0.
   2022-05-09 15:06:39,917 [grpc-default-executor-0] ERROR impl.SnapshotInstallationHandler (SnapshotInstallationHandler.java:installSnapshot(86)) - 2d9f16d2-1d71-4978-9546-00aa402e881d@group-F84A6C219907: installSnapshot failed
   java.lang.IllegalStateException: inProgressInstallSnapshotRequest: 0 is not eligible, firstAvailableLogIndex: 0
           at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
           at org.apache.ratis.server.impl.SnapshotInstallationHandler.notifyStateMachineToInstallSnapshot(SnapshotInstallationHandler.java:287)
           at org.apache.ratis.server.impl.SnapshotInstallationHandler.installSnapshotImpl(SnapshotInstallationHandler.java:115)
           at org.apache.ratis.server.impl.SnapshotInstallationHandler.installSnapshot(SnapshotInstallationHandler.java:84)
           at org.apache.ratis.server.impl.RaftServerImpl.installSnapshot(RaftServerImpl.java:1427)
   ```
   
   Note that this patch fixes the problem, but I'm completely not sure if this is the right way to fix it.
   
   https://issues.apache.org/jira/browse/RATIS-1577
   
   ## How was this patch tested?
   
   ### Regular Ratis CI
   
   https://github.com/adoroszlai/incubator-ratis/actions/runs/2290702895
   
   ### Integration with Ozone
   
   Built local Ratis snapshot:
   
   ```
   ratis_hash=$(git rev-parse --short HEAD)
   ratis_version="2.3.0-${ratis_hash}-SNAPSHOT"
   mvn versions:set -DnewVersion="${ratis_version}"
   mvn -DskipTests clean install
   git reset --hard
   git clean -fd 
   ```
   
   Applied patch to Ozone (needed for compatibility with Ratis `master`):
   
   ```diff
   diff --git hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java
   index 0cb3e6553..6a0a615a3 100644
   --- hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java
   +++ hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java
   @@ -34,9 +34,6 @@
    import org.apache.ozone.test.tag.Flaky;
    
    import static org.apache.hadoop.ozone.OzoneConfigKeys.DFS_RATIS_LEADER_ELECTION_MINIMUM_TIMEOUT_DURATION_KEY;
   -import org.apache.log4j.Level;
   -import org.apache.log4j.Logger;
   -import org.apache.ratis.grpc.client.GrpcClientProtocolService;
    import org.apache.ratis.protocol.ClientId;
    import org.apache.ratis.protocol.GroupInfoReply;
    import org.apache.ratis.protocol.GroupInfoRequest;
   @@ -46,7 +43,9 @@
    import org.junit.jupiter.api.BeforeAll;
    import org.junit.jupiter.api.Test;
    import org.junit.jupiter.api.Timeout;
   +import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
   +import org.slf4j.event.Level;
    
    /**
     * Test pipeline leader information is correctly used.
   @@ -55,7 +54,7 @@
    public class TestRatisPipelineLeader {
      private static MiniOzoneCluster cluster;
      private static OzoneConfiguration conf;
   -  private static final org.slf4j.Logger LOG =
   +  private static final Logger LOG =
          LoggerFactory.getLogger(TestRatisPipelineLeader.class);
    
      @BeforeAll
   @@ -97,9 +96,11 @@ public void testLeaderIdUsedOnFirstCall() throws Exception {
        // Verify client connects to Leader without NotLeaderException
        XceiverClientRatis xceiverClientRatis =
            XceiverClientRatis.newXceiverClientRatis(ratisPipeline, conf);
   -    Logger.getLogger(GrpcClientProtocolService.class).setLevel(Level.DEBUG);
   +    final Logger log = LoggerFactory.getLogger(
   +        "org.apache.ratis.grpc.server.GrpcClientProtocolService");
   +    GenericTestUtils.setLogLevel(log, Level.DEBUG);
        GenericTestUtils.LogCapturer logCapturer =
   -        GenericTestUtils.LogCapturer.captureLogs(GrpcClientProtocolService.LOG);
   +        GenericTestUtils.LogCapturer.captureLogs(log);
        xceiverClientRatis.connect();
        ContainerProtocolCalls.createContainer(xceiverClientRatis, 1L, null);
        logCapturer.stopCapturing();
   diff --git hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/FollowerAppendLogEntryGenerator.java hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/FollowerAppendLogEntryGenerator.java
   index e30b778a1..07d0db800 100644
   --- hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/FollowerAppendLogEntryGenerator.java
   +++ hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/FollowerAppendLogEntryGenerator.java
   @@ -336,7 +336,7 @@ private void configureGroup() throws IOException {
                .build());
        RaftClient client = RaftClient.newBuilder()
            .setClientId(clientId)
   -        .setProperties(new RaftProperties(true))
   +        .setProperties(new RaftProperties())
            .setRaftGroup(group)
            .build();
    
   diff --git hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/LeaderAppendLogEntryGenerator.java hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/LeaderAppendLogEntryGenerator.java
   index 971e6f7e4..8943ca1fa 100644
   --- hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/LeaderAppendLogEntryGenerator.java
   +++ hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/LeaderAppendLogEntryGenerator.java
   @@ -265,7 +265,7 @@ private void configureGroup() throws IOException {
                .build());
        RaftClient client = RaftClient.newBuilder()
            .setClientId(clientId)
   -        .setProperties(new RaftProperties(true))
   +        .setProperties(new RaftProperties())
            .setRaftGroup(group)
            .build();
    
   ```
   
   Ran Ozone test:
   
   ```
   mvn -Dskip.installnpm -Dskip.installnpx -Dskip.installyarn -Dskip.npm -Dskip.npx -Dskip.yarn -DskipShade \
     -am -pl :ozone-integration-test -Dsurefire.fork.timeout=120 -DfailIfNoTests=false -Dtest=TestStorageContainerManagerHA#testAllSCMAreRunning \
     -Dratis.version="$ratis_version" \
     -Dratis.thirdparty.version=1.0.0 -Dgrpc.protobuf-compile.version=3.19.2 -Dnetty.version=4.1.74.Final -Dio.grpc.version=1.44.0 -Dtcnative.version=2.0.48.Final \
     clean test
   ```
   
   ```
   [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.43 s - in org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] adoroszlai commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1121261509

   > Note that this patch fixes the problem, but I'm completely not sure if this is the right way to fix it.
   
   @Xushaohong @szetszwo please take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] adoroszlai commented on a diff in pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #643:
URL: https://github.com/apache/ratis/pull/643#discussion_r869268172


##########
ratis-server/src/main/java/org/apache/ratis/server/impl/SnapshotInstallationHandler.java:
##########
@@ -206,16 +209,16 @@ private InstallSnapshotReplyProto notifyStateMachineToInstallSnapshot(
       state.setLeader(leaderId, "installSnapshot");
       server.updateLastRpcTime(FollowerState.UpdateType.INSTALL_SNAPSHOT_NOTIFICATION);
 
-      if (inProgressInstallSnapshotIndex.compareAndSet(0, firstAvailableLogIndex)) {
+      if (inProgressInstallSnapshotIndex.compareAndSet(INVALID_LOG_INDEX, firstAvailableLogIndex)) {
         LOG.info("{}: Received notification to install snapshot at index {}", getMemberId(), firstAvailableLogIndex);
         // Check if snapshot index is already at par or ahead of the first
         // available log index of the Leader.
         final long snapshotIndex = state.getLog().getSnapshotIndex();
-        if (snapshotIndex + 1 >= firstAvailableLogIndex && firstAvailableLogIndex > 0) {
+        if (snapshotIndex >= firstAvailableLogIndex && firstAvailableLogIndex > INVALID_LOG_INDEX) {

Review Comment:
   Thanks for the explanation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] szetszwo merged pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
szetszwo merged PR #643:
URL: https://github.com/apache/ratis/pull/643


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] adoroszlai commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1122191519

   > If firstAvailable is 0, let's return null?
   
   This in itself fixes the `IllegalStateException`, but follower SCM startup gets stuck.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] szetszwo commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1122066331

   If firstAvailable is 0, let's return null?
   ```
   +++ b/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java
   @@ -588,7 +588,7 @@ public class GrpcLogAppender extends LogAppenderBase {
          // be notified to install a snapshot. Every follower should try to install at least one snapshot during
          // bootstrapping, if available.
          LOG.debug("{}: follower is bootstrapping, notify to install snapshot to {}.", this, firstAvailable);
   -      return firstAvailable;
   +      return firstAvailable.getIndex() > RaftLog.LEAST_VALID_LOG_INDEX? firstAvailable: null;
        }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] szetszwo commented on a diff in pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
szetszwo commented on code in PR #643:
URL: https://github.com/apache/ratis/pull/643#discussion_r869072974


##########
ratis-server/src/main/java/org/apache/ratis/server/impl/SnapshotInstallationHandler.java:
##########
@@ -49,15 +49,18 @@
 import java.util.concurrent.atomic.AtomicLong;
 import java.util.concurrent.atomic.AtomicReference;
 
+import static org.apache.ratis.server.raftlog.RaftLog.INVALID_LOG_INDEX;
+
 class SnapshotInstallationHandler {
   static final Logger LOG = LoggerFactory.getLogger(SnapshotInstallationHandler.class);
 
   private final RaftServerImpl server;
   private final ServerState state;
 
   private final boolean installSnapshotEnabled;
-  private final AtomicLong inProgressInstallSnapshotIndex = new AtomicLong();
-  private final AtomicReference<TermIndex> installedSnapshotTermIndex = new AtomicReference<>(TermIndex.valueOf(0,0));
+  private final AtomicLong inProgressInstallSnapshotIndex = new AtomicLong(INVALID_LOG_INDEX);
+  private final AtomicReference<TermIndex> installedSnapshotTermIndex =
+    new AtomicReference<>(TermIndex.valueOf(0, INVALID_LOG_INDEX));

Review Comment:
   Let's add an INVALID_TERM_INDEX constant.
   ```
     static final TermIndex INVALID_TERM_INDEX = TermIndex.valueOf(0, INVALID_LOG_INDEX);
   ```
   



##########
ratis-server/src/main/java/org/apache/ratis/server/impl/SnapshotInstallationHandler.java:
##########
@@ -206,16 +209,16 @@ private InstallSnapshotReplyProto notifyStateMachineToInstallSnapshot(
       state.setLeader(leaderId, "installSnapshot");
       server.updateLastRpcTime(FollowerState.UpdateType.INSTALL_SNAPSHOT_NOTIFICATION);
 
-      if (inProgressInstallSnapshotIndex.compareAndSet(0, firstAvailableLogIndex)) {
+      if (inProgressInstallSnapshotIndex.compareAndSet(INVALID_LOG_INDEX, firstAvailableLogIndex)) {
         LOG.info("{}: Received notification to install snapshot at index {}", getMemberId(), firstAvailableLogIndex);
         // Check if snapshot index is already at par or ahead of the first
         // available log index of the Leader.
         final long snapshotIndex = state.getLog().getSnapshotIndex();
-        if (snapshotIndex + 1 >= firstAvailableLogIndex && firstAvailableLogIndex > 0) {
+        if (snapshotIndex >= firstAvailableLogIndex && firstAvailableLogIndex > INVALID_LOG_INDEX) {

Review Comment:
   snapshotIndex + 1 seems correct since snapshotIndex is the last index in a snapshot.  Then, leader can send log starting at firstAvailableLogIndex to this server.



##########
ratis-server/src/main/java/org/apache/ratis/server/impl/SnapshotInstallationHandler.java:
##########
@@ -284,31 +287,31 @@ private InstallSnapshotReplyProto notifyStateMachineToInstallSnapshot(
       }
 
       final long inProgressInstallSnapshotIndexValue = getInProgressInstallSnapshotIndex();
-      Preconditions.assertTrue(
-          inProgressInstallSnapshotIndexValue <= firstAvailableLogIndex && inProgressInstallSnapshotIndexValue > 0,
+      Preconditions.assertTrue(inProgressInstallSnapshotIndexValue <= firstAvailableLogIndex
+              && inProgressInstallSnapshotIndexValue > INVALID_LOG_INDEX,
           "inProgressInstallSnapshotRequest: %s is not eligible, firstAvailableLogIndex: %s",
           getInProgressInstallSnapshotIndex(), firstAvailableLogIndex);
 
       // If the snapshot is null or unavailable, return SNAPSHOT_UNAVAILABLE.
       if (isSnapshotNull.compareAndSet(true, false)) {
         LOG.info("{}: InstallSnapshot notification result: {}", getMemberId(),
             InstallSnapshotResult.SNAPSHOT_UNAVAILABLE);
-        inProgressInstallSnapshotIndex.set(0);
+        inProgressInstallSnapshotIndex.set(INVALID_LOG_INDEX);
         return ServerProtoUtils.toInstallSnapshotReplyProto(leaderId, getMemberId(),
             currentTerm, InstallSnapshotResult.SNAPSHOT_UNAVAILABLE, -1);
       }
 
       // If a snapshot has been installed, return SNAPSHOT_INSTALLED with the installed snapshot index and reset
-      // installedSnapshotIndex to (0,0).
+      // installedSnapshotIndex to (0,-1).
       final TermIndex latestInstalledSnapshotTermIndex = this.installedSnapshotTermIndex
-          .getAndSet(TermIndex.valueOf(0,0));
-      if (latestInstalledSnapshotTermIndex.getIndex() > 0) {
+          .getAndSet(TermIndex.valueOf(0, INVALID_LOG_INDEX));
+      if (latestInstalledSnapshotTermIndex.getIndex() > INVALID_LOG_INDEX) {

Review Comment:
   Let's use INVALID_TERM_INDEX.
   ```
         // installedSnapshotTermIndex to INVALID_TERM_INDEX.
         final TermIndex latestInstalledSnapshotTermIndex = this.installedSnapshotTermIndex.getAndSet(INVALID_TERM_INDEX);
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] adoroszlai commented on a diff in pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #643:
URL: https://github.com/apache/ratis/pull/643#discussion_r869175948


##########
ratis-server/src/main/java/org/apache/ratis/server/impl/SnapshotInstallationHandler.java:
##########
@@ -206,16 +209,16 @@ private InstallSnapshotReplyProto notifyStateMachineToInstallSnapshot(
       state.setLeader(leaderId, "installSnapshot");
       server.updateLastRpcTime(FollowerState.UpdateType.INSTALL_SNAPSHOT_NOTIFICATION);
 
-      if (inProgressInstallSnapshotIndex.compareAndSet(0, firstAvailableLogIndex)) {
+      if (inProgressInstallSnapshotIndex.compareAndSet(INVALID_LOG_INDEX, firstAvailableLogIndex)) {
         LOG.info("{}: Received notification to install snapshot at index {}", getMemberId(), firstAvailableLogIndex);
         // Check if snapshot index is already at par or ahead of the first
         // available log index of the Leader.
         final long snapshotIndex = state.getLog().getSnapshotIndex();
-        if (snapshotIndex + 1 >= firstAvailableLogIndex && firstAvailableLogIndex > 0) {
+        if (snapshotIndex >= firstAvailableLogIndex && firstAvailableLogIndex > INVALID_LOG_INDEX) {

Review Comment:
   Do I understand correctly that `snapshotIndex` is the last index in a snapshot that follower already installed, and `firstAvailableLogIndex` is the first index available from leader that would be part of a new snapshot if follower requested?
   
   If so, assume `snapshotIndex = 9` and `firstAvailableLogIndex = 10`.  In this case 9+1 >= 10, hence follower responds with `ALREADY_INSTALLED`.  Does that seem right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] Xushaohong commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
Xushaohong commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1121875913

   As SCM uses bootstrap to set up HA, it goes through this branch.
   ```
   if (isFollowerBootstrapping && !follower.hasAttemptedToInstallSnapshot()) {
         // If the follower is bootstrapping and has not yet installed any snapshot from leader, then the follower should
         // be notified to install a snapshot. Every follower should try to install at least one snapshot during
         // bootstrapping, if available.
         LOG.debug("{}: follower is bootstrapping, notify to install snapshot to {}.", this, firstAvailable);
         return firstAvailable;
       }
   ```
   Meanwhile, the leader is also newly started, which has not truncated the raft log yet. The `firstAvailable` will be 0 here. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] adoroszlai commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1122004205

   Thanks @Xushaohong for taking a look.
   
   > The leader is newly started and has not truncated the raft log yet. The firstAvailable will be 0 here.
   
   Should it wait until truncation before sending the notifyInstallSnapshot message?
   
   > installing a snapshot to 0 is strange
   
   Does the snapshot with index 0 contain any information?  What about later terms (e.g. `t:5, i:0`)?
   
   If it's an empty snapshot, should we short-circuit such snapshot by returning e.g. `ALREADY_INSTALLED`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] Xushaohong commented on a diff in pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
Xushaohong commented on code in PR #643:
URL: https://github.com/apache/ratis/pull/643#discussion_r869187595


##########
ratis-server/src/main/java/org/apache/ratis/server/impl/SnapshotInstallationHandler.java:
##########
@@ -206,16 +209,16 @@ private InstallSnapshotReplyProto notifyStateMachineToInstallSnapshot(
       state.setLeader(leaderId, "installSnapshot");
       server.updateLastRpcTime(FollowerState.UpdateType.INSTALL_SNAPSHOT_NOTIFICATION);
 
-      if (inProgressInstallSnapshotIndex.compareAndSet(0, firstAvailableLogIndex)) {
+      if (inProgressInstallSnapshotIndex.compareAndSet(INVALID_LOG_INDEX, firstAvailableLogIndex)) {
         LOG.info("{}: Received notification to install snapshot at index {}", getMemberId(), firstAvailableLogIndex);
         // Check if snapshot index is already at par or ahead of the first
         // available log index of the Leader.
         final long snapshotIndex = state.getLog().getSnapshotIndex();
-        if (snapshotIndex + 1 >= firstAvailableLogIndex && firstAvailableLogIndex > 0) {
+        if (snapshotIndex >= firstAvailableLogIndex && firstAvailableLogIndex > INVALID_LOG_INDEX) {

Review Comment:
   > Do I understand correctly that `snapshotIndex` is the last index in a snapshot that follower already installed, and `firstAvailableLogIndex` is the first index available from leader that would be part of a new snapshot if follower requested?
   > 
   > If so, assume `snapshotIndex = 9` and `firstAvailableLogIndex = 10`. In this case 9+1 >= 10, hence follower responds with `ALREADY_INSTALLED`. Does that seem right?
   
   I think so. Hope @szetszwo  help confirm it 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] szetszwo commented on a diff in pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
szetszwo commented on code in PR #643:
URL: https://github.com/apache/ratis/pull/643#discussion_r869195343


##########
ratis-server/src/main/java/org/apache/ratis/server/impl/SnapshotInstallationHandler.java:
##########
@@ -206,16 +209,16 @@ private InstallSnapshotReplyProto notifyStateMachineToInstallSnapshot(
       state.setLeader(leaderId, "installSnapshot");
       server.updateLastRpcTime(FollowerState.UpdateType.INSTALL_SNAPSHOT_NOTIFICATION);
 
-      if (inProgressInstallSnapshotIndex.compareAndSet(0, firstAvailableLogIndex)) {
+      if (inProgressInstallSnapshotIndex.compareAndSet(INVALID_LOG_INDEX, firstAvailableLogIndex)) {
         LOG.info("{}: Received notification to install snapshot at index {}", getMemberId(), firstAvailableLogIndex);
         // Check if snapshot index is already at par or ahead of the first
         // available log index of the Leader.
         final long snapshotIndex = state.getLog().getSnapshotIndex();
-        if (snapshotIndex + 1 >= firstAvailableLogIndex && firstAvailableLogIndex > 0) {
+        if (snapshotIndex >= firstAvailableLogIndex && firstAvailableLogIndex > INVALID_LOG_INDEX) {

Review Comment:
   Yes, ALREADY_INSTALLED is correct.
   
   Leader tells firstAvailableLogIndex (=10) to a follower.   The follower checks if it needs to install a snapshot -- if its snapshotIndex < 9, say snapshotIndex == 8, there is a gap between snapshotIndex and firstAvailableLogIndex so that it needs to install a snapshot with index >= 9.  The leader is not able to send the index 9 log entry to the follower because firstAvailableLogIndex is 10.
   
   If the follower's snapshotIndex == 9, there is no gap so that it can tell the leader to send log entires starting at index 10 without installing a snapshot.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] szetszwo commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1122522080

   @adoroszlai , thanks a lot for working on this!
   
   @Xushaohong , thanks for reviewing this change!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] adoroszlai commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1122524766

   Thanks @Xushaohong and @szetszwo for the guidance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [ratis] Xushaohong commented on pull request #643: RATIS-1577. Install snapshot failure

Posted by GitBox <gi...@apache.org>.
Xushaohong commented on PR #643:
URL: https://github.com/apache/ratis/pull/643#issuecomment-1122035694

   > Thanks @Xushaohong for taking a look.
   > 
   > > The leader is newly started and has not truncated the raft log yet. The firstAvailable will be 0 here.
   > 
   > Should it wait until truncation before sending the notifyInstallSnapshot message?
   
   I think it no necessary. The original purpose of https://issues.apache.org/jira/browse/RATIS-1390 seems to trigger an immediate snapshot install for a bootstrapped peer. The written prerequisite is ```The leader sends the snapshot only when some or all of the ratis logs have been purged.```  
   The truth is currently it will trigger a snapshot no matter whether the ratis log has been purged.  In my understanding, it's not bad and also makes sense since such a case only happens at the init stage of the cluster.
   
   
   > > installing a snapshot to 0 is strange
   > Does the snapshot with index 0 contain any information? What about later terms (e.g. `t:5, i:0`)?
   > If it's an empty snapshot, should we short-circuit such snapshot by returning e.g. `ALREADY_INSTALLED`?
   
   Nope.
   The first available index received from the ratis side for OM/SCM/Container is not used, only shown in logs as for information and we obtain the DB checkpoint from the leader for the latest one, whose index could be already updated. 
   For the ozone side, there won't be any empty snapshots. Even before any data is written in the cluster, a snapshot/checkpoint of RDB is non-empty.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org