You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ivandika3 (via GitHub)" <gi...@apache.org> on 2024/10/16 05:40:11 UTC

[PR] HDDS-11585. Add DN Ratis log purge parameters to close pipeline with slow follower [ozone]

ivandika3 opened a new pull request, #7319:
URL: https://github.com/apache/ozone/pull/7319

   ## What changes were proposed in this pull request?
   
   Ozone Ratis pipeline seem to have an indirect mechanism to detect follower with lagging follower index (i.e. "slow follower") through the notifyInstallSnapshot mechanism (since installSnapshot is always disabled).
   
   The idea is that if the leader already purge the logs up to the snapshot index, and the leader's first index is higher than the slow follower's next index (i.e. the log to replicate to slow follower has been purged), the leader will send the notifyInstallSnapshot request to follower and the follower will call StateMachine#notifyInstallSnapshotFromLeader API. 
   
   Datanode implementation of notifyInstallSnapshotFromLeader is to close the pipeline. This indirectly acts as an automatic "slow follower detector" which might be helpful since by default will watch for ALL_COMMITTED (i.e. log index needs to be replicated in all DNs) and will increase write latency considerably.
   
   See the following follower index lag that causes prolonged cluster write degradation that required administrator to close the pipeline manually. Even after the difference of the log index between leader and follower reaches > 1 million, the pipeline is not automatically closed. 
   
   The root cause is because raft.server.log.purge.upto.snapshot.index defaults to false. This means that the leader will not purge the logs until it has been replicated to the slow follower. Therefore, the notifyInstallSnapshot mechanism will never be triggered.
   
   Just like [HDDS-8131](https://issues.apache.org/jira/browse/HDDS-8131), I propose to make raft.server.log.purge.upto.snapshot.index and raft.server.log.purge.preservation.log.num to be configurable. The recommended configuration would be
   - raft.server.log.purge.upto.snapshot.index = true
   - raft.server.log.purge.preservation.log.num = <SLOW_FOLLOWER_THRESHOLD>
   
   Other snapshot configurations such might need to be revisited
   - raft.server.snapshot.auto.trigger.threshold (hdds.ratis.snapshot.threshold) = default is 100,000
   - raft.server.log.purge.gap (hdds.container.ratis.log.purge.gap) = default is 1,000,000
     - This might be too large, we can reduce it to something like 100,000
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-11585
   
   ## How was this patch tested?
   
   Existing tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11585. Add DN Ratis log purge parameters to close pipeline with slow follower [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #7319:
URL: https://github.com/apache/ozone/pull/7319#issuecomment-2630256849

   Closing this for now. 
   
   Automatically closing the pipeline for slow follower might hide the underlying write degradation issue and might cause pipelines to be created and closed frequently which will cause further issues (QUASI_CLOSED containers with different BCSID, unneeded replications).
   
   It's better to scale the datanodes to handle the write degradations or increase the number of pipelines per DN.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11585. Add DN Ratis log purge parameters to close pipeline with slow follower [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 closed pull request #7319: HDDS-11585. Add DN Ratis log purge parameters to close pipeline with slow follower
URL: https://github.com/apache/ozone/pull/7319


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org