[jira] [Created] (HDDS-11585) Add DN Ratis log purge parameters to detect slow follower

["Ivan Andika (Jira)" <ji...@apache.org>]

Ivan Andika created HDDS-11585:
----------------------------------

             Summary: Add DN Ratis log purge parameters to detect slow follower
                 Key: HDDS-11585
                 URL: https://issues.apache.org/jira/browse/HDDS-11585
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Datanode
            Reporter: Ivan Andika
            Assignee: Ivan Andika
         Attachments: image-2024-10-15-11-40-41-618.png, image-2024-10-15-11-56-47-259.png

Ozone Ratis pipeline seem to have indirect mechanism to detect "slow follower" through the notifyInstallSnapshot mechanism.

The idea is that if the leader already purge the logs up to the snapshot index, and the leader's first index is higher than the slow follower's next index (i.e. the log to replicate to slow follower has been purged), the leader will send the notifyInstallSnapshot request to follower and the follower will call StateMachine#notifyInstallSnapshotFromLeader API. 

Datanode implementation of notifyInstallSnapshotFromLeader is to close the pipeline. This indirectly acts as an automatic "slow follower detector" which might be helpful since by default will watch for ALL_COMMITTED (i.e. log index needs to be replicated in all DNs) and will increase write latency considerably.

See the following follower index lag that causes prolonged cluster write degradation that required administrator to close the pipeline manually.

!image-2024-10-15-11-56-47-259.png|width=582,height=134!

Even after the difference of the log index between leader and follower reaches > 1 million, the pipeline is not automatically closed. 

The root cause raft.server.log.purge.upto.snapshot.index default is false. This means that the leader will not purge the logs until it has been replicated to the slow follower. Therefore, the notifyInstallSnapshot mechanism will never be triggered.

Just like HDDS-8131, I propose to make raft.server.log.purge.upto.snapshot.index and raft.server.log.purge.preservation.log.num to be configurable. The recommended configuration would be
 * raft.server.log.purge.upto.snapshot.index = true
 * raft.server.log.purge.preservation.log.num = <SLOW_FOLLOWER_THRESHOLD>

Other snapshot configurations such raft.server.snapshot.auto.trigger.threshold (dfs.ratis.snapshot.threshold) also need to be revisited.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org