[jira] [Assigned] (IGNITE-16675) Need to investigate why initialisation of raft groups could be time-consuming.

["Kirill Gusakov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kirill Gusakov reassigned IGNITE-16675:
---------------------------------------

    Assignee: Kirill Gusakov

> Need to investigate why initialisation of raft groups could be time-consuming. 
> -------------------------------------------------------------------------------
>
>                 Key: IGNITE-16675
>                 URL: https://issues.apache.org/jira/browse/IGNITE-16675
>             Project: Ignite
>          Issue Type: Task
>            Reporter: Mirza Aliev
>            Assignee: Kirill Gusakov
>            Priority: Major
>              Labels: ignite-3
>         Attachments: screenshot-1.png, screenshot-2.png
>
>
> After some investigation that was made under IGNITE-16559 (see the [comment|https://issues.apache.org/jira/browse/IGNITE-16559?focusedCommentId=17495362&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17495362]), we came up with the idea that we could investigate why initialisation of raft groups could be time-consuming.
> We see that init phase of starting raft group contains some time-consuming operations like {{fsync}} or {{RocksDB.open}}
> {noformat}
> 	  at sun.nio.ch.FileDispatcherImpl.force0(FileDispatcherImpl.java:-1)
> 	  at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:82)
> 	  at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:461)
> 	  at org.apache.ignite.raft.jraft.util.Utils.fsync(Utils.java:366)
> 	  at org.apache.ignite.raft.jraft.storage.io.MessageFile.save(MessageFile.java:94)
> 	  at org.apache.ignite.raft.jraft.storage.impl.LocalRaftMetaStorage.save(LocalRaftMetaStorage.java:114)
> 	  at org.apache.ignite.raft.jraft.storage.impl.LocalRaftMetaStorage.setTermAndVotedFor(LocalRaftMetaStorage.java:186)
> 	  at org.apache.ignite.raft.jraft.core.NodeImpl.electSelf(NodeImpl.java:1271)
> 	  at org.apache.ignite.raft.jraft.core.NodeImpl.init(NodeImpl.java:1054)
> 	  at org.apache.ignite.raft.jraft.core.NodeImpl.init(NodeImpl.java:126)
> 	  at org.apache.ignite.raft.jraft.RaftGroupService.start(RaftGroupService.java:108)
> 	  - locked <0x1e42> (a org.apache.ignite.raft.jraft.RaftGroupService)
> 	  at org.apache.ignite.internal.raft.server.impl.JraftServerImpl.startRaftGroup(JraftServerImpl.java:341)
> 	  - locked <0x1deb> (a org.apache.ignite.internal.raft.server.impl.JraftServerImpl)
> 	  at org.apache.ignite.internal.raft.Loza.prepareRaftGroupInternal(Loza.java:193)
> 	  at org.apache.ignite.internal.raft.Loza.prepareRaftGroup(Loza.java:168)
> {noformat}
> {noformat}
> 	  at org.rocksdb.RocksDB.open(RocksDB.java:-1)
> 	  at org.rocksdb.RocksDB.open(RocksDB.java:306)
> 	  at org.apache.ignite.raft.jraft.storage.impl.RocksDBLogStorage.openDB(RocksDBLogStorage.java:308)
> 	  at org.apache.ignite.raft.jraft.storage.impl.RocksDBLogStorage.initAndLoad(RocksDBLogStorage.java:221)
> 	  at org.apache.ignite.raft.jraft.storage.impl.RocksDBLogStorage.init(RocksDBLogStorage.java:198)
> 	  at org.apache.ignite.raft.jraft.storage.impl.RocksDBLogStorage.init(RocksDBLogStorage.java:68)
> 	  at org.apache.ignite.raft.jraft.storage.impl.LogManagerImpl.init(LogManagerImpl.java:183)
> 	  at org.apache.ignite.raft.jraft.storage.impl.LogManagerImpl.init(LogManagerImpl.java:65)
> 	  at org.apache.ignite.raft.jraft.core.NodeImpl.initLogStorage(NodeImpl.java:557)
> 	  at org.apache.ignite.raft.jraft.core.NodeImpl.init(NodeImpl.java:946)
> 	  at org.apache.ignite.raft.jraft.core.NodeImpl.init(NodeImpl.java:126)
> 	  at org.apache.ignite.raft.jraft.RaftGroupService.start(RaftGroupService.java:108)
> 	  - locked <merged>(a org.apache.ignite.raft.jraft.RaftGroupService)
> 	  at org.apache.ignite.internal.raft.server.impl.JraftServerImpl.startRaftGroup(JraftServerImpl.java:341)
> 	  - locked <merged>(a org.apache.ignite.internal.raft.server.impl.JraftServerImpl)
> 	  at org.apache.ignite.internal.raft.Loza.prepareRaftGroupInternal(Loza.java:193)
> 	  at org.apache.ignite.internal.raft.Loza.prepareRaftGroup(Loza.java:168)
> {noformat}
> We made some pre-investigation, we started a raft group on one node 1000 times, red line on the screenshot is a local run, other lines are from TC. Y axis shows milliseconds. X axis represents test attmept. We measured {{org.apache.ignite.raft.jraft.RaftGroupService#start}}. In general, we could see that attempts are stable and they are not time-consuming (in some bad cases, start could last about 1 second. We saw that behavour in TC), but there are some statistical outliers, probably the are related to GC pauses.
>  !screenshot-2.png! 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)