You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ratis.apache.org by Asad Awadia <as...@gmail.com> on 2021/12/22 04:43:16 UTC
Snapshot and log truncation
Looking into snapshotting the statemachine I had a few questions
1. When is the takeSnapshot in the StateMachine function invoked and by whom?
2. Does the snapshot need to be in a single file returned by storage.getSnapshotFile ? If my DB has 5 files I have to merge them into one or something? Any compression done?
3. Is the log truncated after the snapshot? Is that automatic? Any frequency configuration required to be set?
4. When I backup my DB - what parts of the snapshot should I backup alongside my own DB files?
Thank you again for your help
Re: Snapshot and log truncation
Posted by Tsz Wo Sze <sz...@gmail.com>.
> Is there a properties/options object that I have to pass ratis?
Yes. When building a RaftServer/RaftClient, we may pass a RaftProperties
object.
> Are these set via environment variables?
Ratis itself currently won't pick up conf properties from environment
variables. Applications may (1) get the values from environment variables,
(2) set values to a RaftProperties and then (3) pass the RaftProperties to
build a RaftServer/RaftClient.
Thanks for your questions!
Tsz-Wo
On Wed, Jan 5, 2022 at 10:45 PM Asad Awadia <as...@gmail.com> wrote:
> Are these set via environment variables? Is there a properties/options
> object that I have to pass ratis?
>
> Regards,
> Asad
> ------------------------------
> *From:* Tsz Wo Sze <sz...@gmail.com>
> *Sent:* Tuesday, December 28, 2021 5:53:10 AM
> *To:* user@ratis.apache.org <us...@ratis.apache.org>
> *Subject:* Re: Snapshot and log truncation
>
> 1. The StateMachineUpdater is a thread to apply log transactions to the
> state machine and take snapshots (by
> calling StateMachine.applyTransaction(..)/takeSnapshot()). If
> snapshot-auto-trigger is enabled, it will take a snapshot when the number
> of transactions hits the configured threshold.
> - raft.server.snapshot.auto.trigger.enabled (boolean, default=false)
> - raft.server.snapshot.auto.trigger.threshold (long, default=400000)
>
> 2. No, it can be multiple files. The SnapshotInfo interface supports a
> list of files. An implementation is FileListSnapshotInfo.
>
> 3. The server will determine if the log transactions can be purged. There
> are two conf properties:
>
> - raft.server.log.purge.upto.snapshot.index (boolean, default=false)
> Once a snapshot has been taken at index i, try purging all the
> transactions with index up to i.
>
> - raft.server.log.purge.gap (int, default=1024)
> The gap between two purge operations to avoid executing the purge
> operations too frequently. E.g. Suppose the purge.gap is 1024. If a purge
> happened at log index 100, the next purge will NOT happen until log index
> 1124.
>
> 4. The latest snapshot and all the log transactions after it.
>
> Thanks.
> Tsz-Wo
>
>
> On Wed, Dec 22, 2021 at 12:43 PM Asad Awadia <as...@gmail.com> wrote:
>
> Looking into snapshotting the statemachine I had a few questions
>
> 1. When is the takeSnapshot in the StateMachine function invoked and by
> whom?
>
> 2. Does the snapshot need to be in a single file returned by
> storage.getSnapshotFile ? If my DB has 5 files I have to merge them into
> one or something? Any compression done?
>
> 3. Is the log truncated after the snapshot? Is that automatic? Any
> frequency configuration required to be set?
>
> 4. When I backup my DB - what parts of the snapshot should I backup
> alongside my own DB files?
>
>
> Thank you again for your help
>
>
Re: Snapshot and log truncation
Posted by Asad Awadia <as...@gmail.com>.
Are these set via environment variables? Is there a properties/options object that I have to pass ratis?
Regards,
Asad
________________________________
From: Tsz Wo Sze <sz...@gmail.com>
Sent: Tuesday, December 28, 2021 5:53:10 AM
To: user@ratis.apache.org <us...@ratis.apache.org>
Subject: Re: Snapshot and log truncation
1. The StateMachineUpdater is a thread to apply log transactions to the state machine and take snapshots (by calling StateMachine.applyTransaction(..)/takeSnapshot()). If snapshot-auto-trigger is enabled, it will take a snapshot when the number of transactions hits the configured threshold.
- raft.server.snapshot.auto.trigger.enabled (boolean, default=false)
- raft.server.snapshot.auto.trigger.threshold (long, default=400000)
2. No, it can be multiple files. The SnapshotInfo interface supports a list of files. An implementation is FileListSnapshotInfo.
3. The server will determine if the log transactions can be purged. There are two conf properties:
- raft.server.log.purge.upto.snapshot.index (boolean, default=false)
Once a snapshot has been taken at index i, try purging all the transactions with index up to i.
- raft.server.log.purge.gap (int, default=1024)
The gap between two purge operations to avoid executing the purge operations too frequently. E.g. Suppose the purge.gap is 1024. If a purge happened at log index 100, the next purge will NOT happen until log index 1124.
4. The latest snapshot and all the log transactions after it.
Thanks.
Tsz-Wo
On Wed, Dec 22, 2021 at 12:43 PM Asad Awadia <as...@gmail.com>> wrote:
Looking into snapshotting the statemachine I had a few questions
1. When is the takeSnapshot in the StateMachine function invoked and by whom?
2. Does the snapshot need to be in a single file returned by storage.getSnapshotFile ? If my DB has 5 files I have to merge them into one or something? Any compression done?
3. Is the log truncated after the snapshot? Is that automatic? Any frequency configuration required to be set?
4. When I backup my DB - what parts of the snapshot should I backup alongside my own DB files?
Thank you again for your help
Re: Snapshot and log truncation
Posted by Tsz Wo Sze <sz...@gmail.com>.
1. The StateMachineUpdater is a thread to apply log transactions to the
state machine and take snapshots (by
calling StateMachine.applyTransaction(..)/takeSnapshot()). If
snapshot-auto-trigger is enabled, it will take a snapshot when the number
of transactions hits the configured threshold.
- raft.server.snapshot.auto.trigger.enabled (boolean, default=false)
- raft.server.snapshot.auto.trigger.threshold (long, default=400000)
2. No, it can be multiple files. The SnapshotInfo interface supports a
list of files. An implementation is FileListSnapshotInfo.
3. The server will determine if the log transactions can be purged. There
are two conf properties:
- raft.server.log.purge.upto.snapshot.index (boolean, default=false)
Once a snapshot has been taken at index i, try purging all the transactions
with index up to i.
- raft.server.log.purge.gap (int, default=1024)
The gap between two purge operations to avoid executing the purge
operations too frequently. E.g. Suppose the purge.gap is 1024. If a purge
happened at log index 100, the next purge will NOT happen until log index
1124.
4. The latest snapshot and all the log transactions after it.
Thanks.
Tsz-Wo
On Wed, Dec 22, 2021 at 12:43 PM Asad Awadia <as...@gmail.com> wrote:
> Looking into snapshotting the statemachine I had a few questions
>
> 1. When is the takeSnapshot in the StateMachine function invoked and by
> whom?
>
> 2. Does the snapshot need to be in a single file returned by
> storage.getSnapshotFile ? If my DB has 5 files I have to merge them into
> one or something? Any compression done?
>
> 3. Is the log truncated after the snapshot? Is that automatic? Any
> frequency configuration required to be set?
>
> 4. When I backup my DB - what parts of the snapshot should I backup
> alongside my own DB files?
>
>
> Thank you again for your help
>