You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Devesh Kumar Singh (Jira)" <ji...@apache.org> on 2026/04/21 10:19:00 UTC

[jira] [Created] (HDDS-15068) OM start failed with exception: java.lang.IllegalStateException: gap between entry term

Devesh Kumar Singh created HDDS-15068:
-----------------------------------------

             Summary: OM start failed with exception: java.lang.IllegalStateException: gap between entry term
                 Key: HDDS-15068
                 URL: https://issues.apache.org/jira/browse/HDDS-15068
             Project: Apache Ozone
          Issue Type: Task
          Components: Ozone Manager
    Affects Versions: 2.1.0
            Reporter: Devesh Kumar Singh


While creating bulk keys through freon, hit the below stack trace error. Freon was running smooth with all OMs up, but after 1 hour of freon running, one of the follower OM went down and I found exactly the same stack trace, so I thought of running the command :

{code:java}
ozone debug ratislogparser om --segment-path=log_inprogress_890723
{code}


and after all parsing, it failed with same below error.


{code:java}
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between entry term: 11
index: 975728
metadataEntry {
  commitIndex: 975726
}
 and entry term: 11
index: 1251979
metadataEntry {
  commitIndex: 1251977
}

	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
	at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
	at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
	at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalStateException: gap between entry term: 11
index: 975728
metadataEntry {
  commitIndex: 975726
}
 and entry term: 11
index: 1251979
metadataEntry {
  commitIndex: 1251977
}

	at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
	at org.apache.ratis.server.raftlog.segmented.LogSegment.readSegmentFile(LogSegment.java:133)
	at org.apache.ratis.server.raftlog.segmented.LogSegment.loadSegment(LogSegment.java:164)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.loadSegment(SegmentedRaftLogCache.java:381)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:241)
	at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.openImpl(SegmentedRaftLog.java:214)
	at org.apache.ratis.server.raftlog.RaftLogBase.open(RaftLogBase.java:251)
	at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:239)
	at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:220)
	at org.apache.ratis.server.impl.ServerState.lambda$new$5(ServerState.java:161)
	at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
	at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:177)
	at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:338)
	at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:188)
{code}


{code:java}
[root@<cluster>cluster> current]# ls -larht
total 65M
-rw-r--r-- 1 hdfs hdfs   54 Apr 19 16:19 raft-meta
-rw-r--r-- 1 hdfs hdfs  186 Apr 19 16:19 raft-meta.conf
drwxr-xr-x 2 hdfs hdfs 4.0K Apr 19 17:28 .
-rw-r--r-- 1 hdfs hdfs  64M Apr 19 17:28 log_inprogress_890723
drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
[root@<cluster> current]#

[root@<cluster> sm]# pwd
/var/lib/hadoop-ozone/om/ratis/5341b4ef-bcd6-3183-aa83-a6bf76f23ef4/sm
[root@<cluster> sm]# ls -larht
total 8.0K
drwxr-xr-x 2 hdfs hdfs 4.0K Apr 17 06:57 .
drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
[root@<cluster> sm]#
{code}


OM follower is failing to start. Logs for raft folder and OM follower logs are attached. 

cc: [~erose] [~Sammi] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org