[jira] [Commented] (IOTDB-918) [Distributed] Bind TsFiles with log index to accelerate snapshots

["Houliang Qi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IOTDB-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211594#comment-17211594 ] 

Houliang Qi commented on IOTDB-918:
-----------------------------------

Good idea, this will reduce the time taken by the snapshot and speed up the snapshot task.

I think we can add the commit log index to the tsFileResource when close one TsFile, and when a follower request a snapshot, the leader needs to collects the TsFiles that need to be sent to the follower, with the commit log index in the TsFile resource, it's easy to filter the TsFile that do not need to be collected.

BTW, according to my understanding, the commit log index can only indicate that the data in the TsFile will not exceed the commit log index, but it cannot guarantee that the data smaller than the commit log index must exist in the TsFile. Is that the truth?

> [Distributed] Bind TsFiles with log index to accelerate snapshots
> -----------------------------------------------------------------
>
>                 Key: IOTDB-918
>                 URL: https://issues.apache.org/jira/browse/IOTDB-918
>             Project: Apache IoTDB
>          Issue Type: Improvement
>          Components: Core/Cluster
>            Reporter: Tian Jiang
>            Priority: Major
>              Labels: snapshot
>
> During snapshot catch-up, the leader will collect information of all TsFiles as RemoteTsFileResources, and send them to the followers. The follower will check if it has the file and decide whether to pull it from the leader.
> To be smarter, the leader may be able to know that a TsFile of a certain version only contains data precedes to a certain log index, and the leader can avoid sending files that would be surely on the receiver side. 
> For example, when a TsFile `3.tsfile` is closed and the current committed log index is 100, the leader knows for sure that the file will not contain data corresponding to logs whose indexes are more than 100. When a follower whose committed log index exceeds 100 applies for a snapshot, the leader can skip `3.tsfile` since the follower must have it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)