You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Devesh Kumar Singh (Jira)" <ji...@apache.org> on 2026/04/21 10:19:00 UTC
[jira] [Created] (HDDS-15068) OM start failed with exception: java.lang.IllegalStateException: gap between entry term
Devesh Kumar Singh created HDDS-15068:
-----------------------------------------
Summary: OM start failed with exception: java.lang.IllegalStateException: gap between entry term
Key: HDDS-15068
URL: https://issues.apache.org/jira/browse/HDDS-15068
Project: Apache Ozone
Issue Type: Task
Components: Ozone Manager
Affects Versions: 2.1.0
Reporter: Devesh Kumar Singh
While creating bulk keys through freon, hit the below stack trace error. Freon was running smooth with all OMs up, but after 1 hour of freon running, one of the follower OM went down and I found exactly the same stack trace, so I thought of running the command :
{code:java}
ozone debug ratislogparser om --segment-path=log_inprogress_890723
{code}
and after all parsing, it failed with same below error.
{code:java}
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap between entry term: 11
index: 975728
metadataEntry {
commitIndex: 975726
}
and entry term: 11
index: 1251979
metadataEntry {
commitIndex: 1251977
}
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
at java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
at org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalStateException: gap between entry term: 11
index: 975728
metadataEntry {
commitIndex: 975726
}
and entry term: 11
index: 1251979
metadataEntry {
commitIndex: 1251977
}
at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
at org.apache.ratis.server.raftlog.segmented.LogSegment.readSegmentFile(LogSegment.java:133)
at org.apache.ratis.server.raftlog.segmented.LogSegment.loadSegment(LogSegment.java:164)
at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.loadSegment(SegmentedRaftLogCache.java:381)
at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:241)
at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.openImpl(SegmentedRaftLog.java:214)
at org.apache.ratis.server.raftlog.RaftLogBase.open(RaftLogBase.java:251)
at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:239)
at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:220)
at org.apache.ratis.server.impl.ServerState.lambda$new$5(ServerState.java:161)
at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:177)
at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:338)
at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:188)
{code}
{code:java}
[root@<cluster>cluster> current]# ls -larht
total 65M
-rw-r--r-- 1 hdfs hdfs 54 Apr 19 16:19 raft-meta
-rw-r--r-- 1 hdfs hdfs 186 Apr 19 16:19 raft-meta.conf
drwxr-xr-x 2 hdfs hdfs 4.0K Apr 19 17:28 .
-rw-r--r-- 1 hdfs hdfs 64M Apr 19 17:28 log_inprogress_890723
drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
[root@<cluster> current]#
[root@<cluster> sm]# pwd
/var/lib/hadoop-ozone/om/ratis/5341b4ef-bcd6-3183-aa83-a6bf76f23ef4/sm
[root@<cluster> sm]# ls -larht
total 8.0K
drwxr-xr-x 2 hdfs hdfs 4.0K Apr 17 06:57 .
drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
[root@<cluster> sm]#
{code}
OM follower is failing to start. Logs for raft folder and OM follower logs are attached.
cc: [~erose] [~Sammi]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org