[jira] [Created] (IOTDB-562) Apache IoTDB raft log persistence in the distributed version

["Tian Jiang (Jira)" <ji...@apache.org>]

Tian Jiang created IOTDB-562:
--------------------------------

             Summary: Apache IoTDB raft log persistence in the distributed version
                 Key: IOTDB-562
                 URL: https://issues.apache.org/jira/browse/IOTDB-562
             Project: Apache IoTDB
          Issue Type: Improvement
          Components: Core/Cluster
            Reporter: Tian Jiang
            Assignee: Kaifeng Xue


IoTDB is a highly efficient time series database, which supports high-speed query process, including aggregation query.

Currently, IoTDB has supported shared-nothing cluster which using raft mechanism and raft logs to communicate among all nodes. So raft logs are very important in communication, consistency keeping, and fail-over.

However, the current logs are just stored in memory which means raft logs will lost when the nodes down and then recover. Secondly, the raft logs may overlap with current WAL which means we may do some unnecessary log writing works.

So there are two improvements about raft logs need to be done:

1. Store the raft logs in a durable material such as a disk. You need to design a serializable form of logs and then put them to disk.

2. Find a way of using raft logs in the IoTDB recovery process. That's means we just write raft logs rather than both raft logs and WAL. This will avoid some unnecessary log writing works and improve insertion performance.

This proposal is mainly for improving raft logs in clustered IOTDB. Besides, if we can let the summary info be more useful, it could be better.

Notice that the premise is that the raft logs writing process should not be slow down too much. That means the serializable form should be high efficiency enough.

You should know:
• IoTDB cluster structure
• IoTDB WAL
• IoTDB insertion process
• Raft
• Java 

difficulty: Major
mentors: 
jt2594838@163.com



--
This message was sent by Atlassian Jira
(v8.3.4#803005)