[jira] [Created] (IOTDB-4429) [ ratis ]java.lang.IllegalStateException Found a gap between logs ,failed to start datanode

["刘珍 (Jira)" <ji...@apache.org>]

刘珍 created IOTDB-4429:
-------------------------

             Summary: [ ratis ]java.lang.IllegalStateException  Found a gap between logs ,failed to start datanode
                 Key: IOTDB-4429
                 URL: https://issues.apache.org/jira/browse/IOTDB-4429
             Project: Apache IoTDB
          Issue Type: Bug
          Components: mpp-cluster
    Affects Versions: 0.14.0-SNAPSHOT
            Reporter: 刘珍
            Assignee: Song Ziyang


master_0914_ratis ， 基于ratis代码打包：
3C3D3副本，bm写入完成，（12小时后）查询序列的记录数正确。
3台机器手动执行 killall -u current_user   ，
重启启动集群，datanode启动失败：
2022-09-16 10:05:29,073 [main] ERROR o.a.i.c.ServerCommandLine:63 - Failed to execute system command
java.util.concurrent.CompletionException: java.lang.IllegalStateException: Found a gap between logs: the last log segment log-518339_518817 ended at 518817 but the next log segment log-614837_615559 started at 614837
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
        at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1298)
        at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284)
        at java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1034)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
        at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:176)
        at org.apache.ratis.util.ConcurrentUtils.lambda$null$3(ConcurrentUtils.java:165)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Found a gap between logs: the last log segment log-518339_518817 ended at 518817 but the next log segment log-614837_615559 started at 614837
        at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:73)
        at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.validateAdding(SegmentedRaftLogCache.java:430)
        at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.addSegment(SegmentedRaftLogCache.java:437)
        at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.loadSegment(SegmentedRaftLogCache.java:390)
        at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:242)
        at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.openImpl(SegmentedRaftLog.java:215)
        at org.apache.ratis.server.raftlog.RaftLogBase.open(RaftLogBase.java:251)
        at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:186)
        at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:166)
        at org.apache.ratis.server.impl.ServerState.lambda$new$5(ServerState.java:130)
        at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
        at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:147)
        at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:326)
        at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:173)
        ... 4 common frames omitted

测试环境
1. 192.168.130.3/4/5 16cpu32GB
bm在ip2

2. bm配置见附件
运行时间 10.36小时。
全部数据写入成功。

3. （12小时后）select count(s_89) from root.** align by device ;
1600个序列的count值 均= 50000

4. 3台机器，手动输入cmd  killall -u iotdb
echo 3 > /proc/sys/vm/drop_caches

5. 启动ConfigNode 成功

6. 启动DataDode失败



--
This message was sent by Atlassian Jira
(v8.20.10#820010)