[DISCUSS] enhance current installSnapshot flexibility

[William Song <sz...@163.com>]

Hi,

Ratis currently provides two options when it comes to install snapshot to a new follower.

1. By default, leader installs snapshot to follower.
2. Use notification to let follower pull the snapshot from leader.

While the first option is easy to use and requires minimal code, it has some inconveniences:

1. It has restrictions on snapshot layout and does not support the snapshot to be placed outside statemachine dir, as discussed in previous thread https://lists.apache.org/thread/yw9dq78o5hxokw7s8m69ov9o6k1p0pn5 <https://lists.apache.org/thread/yw9dq78o5hxokw7s8m69ov9o6k1p0pn5>.
2. It allows only one snapshot version, and RetentionPolicy won't work.
3. It does not support features like incremental snapshot.

Therefore, Ozone/Alluxio choose to implement InstallSnapshotNotification and let StateMachine take full responsibility. While the second method provides more flexibility, it requires more implementation efforts like snapshot chunking / validating / transferring.

In IoTDB, we also decide to implement InstallSnapshotNotification for our DataStateMachine. However, we still want to use the default installSnapshot method for SchemaStateMachine and PartitionStateMachine.

Therefore, Maybe we can enhance the default installSnapshot methods’ flexibility so that it can cover the needs of most simple scenarios? The goal of this enhancement are:
1. For *most* scenarios, the default method installSnapshot shall be enough and applications shall not bother to implement the complex snapshot mechanism.
2. If the application have very specific requirements, they shall implement their own InstallSnapshotNotification.


To begin with, I think we should support to let state machine decide the snapshot install path. Via this feature, statemachine data and ratis data can be totally separated. I’ve submitted a patch in 
   https://issues.apache.org/jira/secure/attachment/13051659/idea.patch <https://issues.apache.org/jira/secure/attachment/13051659/idea.patch> Any suggestions would be much appreciated!


BTW, I think we should also include the descriptions of installSnapshot / installSnapshotNotification in some kind of Developer’s Guide. NotifyInstallSnapshotFromLeader sits in FollowerApi and it’s hard for a fresh hand to figure out its exact usage and all the above nuances I mentioned. It would save me a lot of time if I know this option in the first place. 
I searched the code repo and there’s no developer's guide for now, and I would like to help to create one.


Thanks for your time. What do you think?

Regards,
William

Re: [DISCUSS] enhance current installSnapshot flexibility

[Tsz Wo Sze <sz...@gmail.com>]

Hi William,

Thanks a lot for working on the InstallSnapshot improvements!  The idea is
great.  It is going to be very useful.  Let's start with letting state
machine decide the snapshot install path.

It is also good to include more documentation.

Tsz-Wo


On Tue, Nov 1, 2022 at 12:59 PM William Song <sz...@163.com> wrote:

> Hi,
>
> Ratis currently provides two options when it comes to install snapshot to
> a new follower.
>
> 1. By default, leader installs snapshot to follower.
> 2. Use notification to let follower pull the snapshot from leader.
>
> While the first option is easy to use and requires minimal code, it has
> some inconveniences:
>
> 1. It has restrictions on snapshot layout and does not support the
> snapshot to be placed outside statemachine dir, as discussed in previous
> thread https://lists.apache.org/thread/yw9dq78o5hxokw7s8m69ov9o6k1p0pn5 <
> https://lists.apache.org/thread/yw9dq78o5hxokw7s8m69ov9o6k1p0pn5>.
> 2. It allows only one snapshot version, and RetentionPolicy won't work.
> 3. It does not support features like incremental snapshot.
>
> Therefore, Ozone/Alluxio choose to implement InstallSnapshotNotification
> and let StateMachine take full responsibility. While the second method
> provides more flexibility, it requires more implementation efforts like
> snapshot chunking / validating / transferring.
>
> In IoTDB, we also decide to implement InstallSnapshotNotification for our
> DataStateMachine. However, we still want to use the default installSnapshot
> method for SchemaStateMachine and PartitionStateMachine.
>
> Therefore, Maybe we can enhance the default installSnapshot methods’
> flexibility so that it can cover the needs of most simple scenarios? The
> goal of this enhancement are:
> 1. For *most* scenarios, the default method installSnapshot shall be
> enough and applications shall not bother to implement the complex snapshot
> mechanism.
> 2. If the application have very specific requirements, they shall
> implement their own InstallSnapshotNotification.
>
>
> To begin with, I think we should support to let state machine decide the
> snapshot install path. Via this feature, statemachine data and ratis data
> can be totally separated. I’ve submitted a patch in
>    https://issues.apache.org/jira/secure/attachment/13051659/idea.patch <
> https://issues.apache.org/jira/secure/attachment/13051659/idea.patch> Any
> suggestions would be much appreciated!
>
>
> BTW, I think we should also include the descriptions of installSnapshot /
> installSnapshotNotification in some kind of Developer’s Guide.
> NotifyInstallSnapshotFromLeader sits in FollowerApi and it’s hard for a
> fresh hand to figure out its exact usage and all the above nuances I
> mentioned. It would save me a lot of time if I know this option in the
> first place.
> I searched the code repo and there’s no developer's guide for now, and I
> would like to help to create one.
>
>
> Thanks for your time. What do you think?
>
> Regards,
> William
>
>
>
>
>
>