You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Song Ziyang (Jira)" <ji...@apache.org> on 2022/09/16 12:00:00 UTC
[jira] [Commented] (IOTDB-4429) [ ratis ]java.lang.IllegalStateException Found a gap between logs ,failed to start datanode
[ https://issues.apache.org/jira/browse/IOTDB-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605786#comment-17605786 ]
Song Ziyang commented on IOTDB-4429:
------------------------------------
There is a full snapshot for group11, therefore, raft log is purged. However, the raft is not fully purged, and there is the gap.
SnapshotLoader cannot find the snapshot.log, so group11 has to recover from Raft Log. The gap causes the above exceptions.
Please take a look [~ThuLiuxuxin]
> [ ratis ]java.lang.IllegalStateException Found a gap between logs ,failed to start datanode
> --------------------------------------------------------------------------------------------
>
> Key: IOTDB-4429
> URL: https://issues.apache.org/jira/browse/IOTDB-4429
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
> Affects Versions: 0.14.0-SNAPSHOT
> Reporter: 刘珍
> Assignee: Song Ziyang
> Priority: Major
> Attachments: iotdb_4429.conf
>
>
> master_0914_ratis ， 基于ratis代码打包：
> 3C3D3副本，bm写入完成，（12小时后）查询序列的记录数正确。
> 3台机器手动执行 killall -u current_user ，
> 重启启动集群，{color:#DE350B}*datanode启动失败*{color}：
> 2022-09-16 10:05:29,073 [main] ERROR o.a.i.c.ServerCommandLine:63 - Failed to execute system command
> java.util.concurrent.CompletionException: java.lang.IllegalStateException: Found a gap between logs: the last log segment log-518339_518817 ended at 518817 but the next log segment log-614837_615559 started at 614837
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1298)
> at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284)
> at java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1034)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:176)
> at org.apache.ratis.util.ConcurrentUtils.lambda$null$3(ConcurrentUtils.java:165)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {color:#DE350B}*Caused by: java.lang.IllegalStateException: Found a gap between logs: the last log segment log-518339_518817 ended at 518817 but the next log segment log-614837_615559 started at 614837*{color}
> at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:73)
> at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.validateAdding(SegmentedRaftLogCache.java:430)
> at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.addSegment(SegmentedRaftLogCache.java:437)
> at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.loadSegment(SegmentedRaftLogCache.java:390)
> at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:242)
> at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.openImpl(SegmentedRaftLog.java:215)
> at org.apache.ratis.server.raftlog.RaftLogBase.open(RaftLogBase.java:251)
> at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:186)
> at org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:166)
> at org.apache.ratis.server.impl.ServerState.lambda$new$5(ServerState.java:130)
> at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
> at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:147)
> at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:326)
> at org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:173)
> ... 4 common frames omitted
> 测试环境
> 1. 192.168.130.3/4/5 16cpu32GB
> bm在ip2
> 数据库路径：
> /data2/cluster_test/master_0914_ratis
> 2. bm配置见附件
> 运行时间 10.36小时。
> 全部数据写入成功。
> 3. （12小时后）select count(s_89) from root.** align by device ;
> 1600个序列的count值 均= 50000
> 4. 3台机器，手动输入cmd killall -u iotdb
> echo 3 > /proc/sys/vm/drop_caches
> 5. 启动ConfigNode 成功
> 6. 启动DataDode失败
--
This message was sent by Atlassian Jira
(v8.20.10#820010)