You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/03/18 09:44:00 UTC
[jira] [Updated] (HDDS-10546) OM startup failure as leader is not getting ready
[ https://issues.apache.org/jira/browse/HDDS-10546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-10546:
----------------------------------
Labels: pull-request-available (was: )
> OM startup failure as leader is not getting ready
> -------------------------------------------------
>
> Key: HDDS-10546
> URL: https://issues.apache.org/jira/browse/HDDS-10546
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.5.0
> Reporter: Sumit Agrawal
> Assignee: Sumit Agrawal
> Priority: Critical
> Labels: pull-request-available
>
> As fix done to simplify OMStateMachine with HDDS-10026, a issue induced.
> Found below log, that leader is selected but is not ready, and further some appendEntry is failing, so its not getting ready.
>
> {code:java}
> 2024-02-22 04:01:34,446 INFO [om1546336109-server-thread2]-org.apache.ratis.server.RaftServer$Division: om1546336109@group-5A2AC78077C7: change Leader from null to om1546336117 at term 2 for appendEntries, leader elected after 2618ms2024-02-22 04:01:34,454 INFO [om1546336109-server-thread1]-org.apache.ratis.server.RaftServer$Division: om1546336109@group-5A2AC78077C7: Failed appendEntries as the first entry (index 0) already exists (snapshotIndex: 0, commitIndex: 0)2024-02-22 04:01:34,484 INFO [om1546336109-server-thread1]-org.apache.ratis.server.RaftServer$Division: om1546336109@group-5A2AC78077C7: inconsistency entries. Reply:om1546336117<-om1546336109#1:FAIL-t2,INCONSISTENCY,nextIndex=1,followerCommit=0,matchIndex=-12024-02-22 04:01:34,523 INFO [om1546336109-server-thread2]-org.apache.ratis.server.RaftServer$Division: om1546336109@group-5A2AC78077C7: Failed appendEntries as previous log entry ((t:2, i:1)) is not found2024-02-22 04:01:34,523 INFO [om1546336109-server-thread2]-org.apache.ratis.server.RaftServer$Division: om1546336109@group-5A2AC78077C7: inconsistency entries. Reply:om1546336117<-om1546336109#2:FAIL-t2,INCONSISTENCY,nextIndex=1,followerCommit=0,matchIndex=-12024-02-22 04:01:34,531 INFO [om1546336109-server-thread2]-org.apache.ratis.server.RaftServer$Division: om1546336109@group-5A2AC78077C7: set configuration 1: peers:[om1546336109|rpc:ccycloud-2.quasar-xrskyv.root.comops.site:9872|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER, om1546336117|rpc:ccycloud-5.quasar-xrskyv.root.comops.site:9872|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER, om1546336113|rpc:ccycloud-7.quasar-xrskyv.root.comops.site:9872|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER]|listeners:[], old=null2024-02-22 04:01:34,538 INFO [om1546336109-server-thread2]-org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker: om1546336109@group-5A2AC78077C7-SegmentedRaftLogWorker: Rolling segment log-0_0 to index:0 {code}
>
>
> Reason:
> Raft side, LeaderStateImpl.java, leader being ready depends on LastAppliedIndex to be updated.
>
> {code:java}
> boolean isReady() {
> return server.getState().getLastAppliedIndex() >= placeHolderIndex;
> } {code}
>
>
> t with HDDS-10026, its getting updated only during takeSnapshot(). As analysis from logs,
>
> {code:java}
> 2024-03-17 16:33:22,573 INFO [om1546336117@group-350E5C994683-StateMachineUpdater]-org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: applied = (t:0, i:~)2024-03-17 16:33:22,573 INFO [om1546336117@group-350E5C994683-StateMachineUpdater]-org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: skipped = -12024-03-17 16:33:22,573 INFO [om1546336117@group-350E5C994683-StateMachineUpdater]-org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: notified = (t:1, i:0)2024-03-17 16:33:22,573 INFO [om1546336117@group-350E5C994683-StateMachineUpdater]-org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: snapshot = (t:1, i:0) {code}
> And later it just stop, no further update.
>
>
> With just updating lastAppliedTermIndex when notified with termIndex from ratis, it started working.
>
> So as solution,
> We need update logic,
> * keep updating lastAppliedTermIndex when notified termIndex is just ascending order (diff is "1")
> * keep last sequence of skipIndex to notifiedIndex, and update if dbUpdate index and skipIndex diff = 1.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org