You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefan Miklosovic (Jira)" <ji...@apache.org> on 2021/09/03 10:03:00 UTC
[jira] [Comment Edited] (CASSANDRA-16911) Remove ephemeral snapshot
marker file and introduce a flag to SnapshotManifest
[ https://issues.apache.org/jira/browse/CASSANDRA-16911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409411#comment-17409411 ]
Stefan Miklosovic edited comment on CASSANDRA-16911 at 9/3/21, 10:02 AM:
-------------------------------------------------------------------------
One not so obvious consequence of this patch is that, until now, there was a "scrubbing" process as part of startup checks which scanned the directories and removed these ephemeral snapshots when that marker file was found.
Since we are using SnapshotManager from now on (or at least we try to drift towards that), it is too soon to involve SnapshotManager at this very early stages of a node boot sequence. The stuff it needs to scan directories for snapshots (iterating over all Keyspaces and so on) is just not there yet.
In this new implementation, we indeed get rid of ephemerals too, but it is done a little bit later (still part of the booting sequence), just after all is initialised properly for SnapshotManager. Removal of ephemerals is done as one-time job and SnapshotManager is started in such a way that periodic checking for expired snapshots is not in effect yet. We are resuming this later in the booting sequence which will be obvious by a respective reader of the patch.
Also, this patch also introduces an interesting problem on upgrade path; imagine this scenario.
# I am running 4.0 where an ephemeral snapshot is taken and it is stored on disk with ephemeral marker file
# this node on 4.0 is turned off, so ephemeral snapshot still lives on disk because it is meant to be removed on next startup
# I upgrade this node to 4.1 (or whatever version this change will be in), but now we are expecting to have SnapshotManifest
# Since there is a manifest file from times when ephemeral flag was not introduced yet and marker file taken into account anymore, by creating SnapshotManifest, it will be evaluated as that snapshot is _not_ ephemeral
# Since it is not ephemeral anymore, we just "promoted" a snapshot from ephemeral to normal one which will be never removed
I think that this problem has to be addressed by writing ephemeral marker files, as it is done now, in paralel with introducing this flag into SnapshotManifest so once a user is on 4.0 and he wants to upgrade to 4.1, ephemeral marker files will be still taken into account and removed on the startup, it is just that any new ephemeral snapshot will not have this marker file and flag in the manifest will b persisted instead of that.
In the next version, like 4.2, we will get rid of marker file logic completely because 4.1 will create all ephemeral snapshots with a flag in the manifest only.
was (Author: stefan.miklosovic):
One not so obvious consequence of this patch is that, until now, there was a "scrubbing" process as part of startup checks which scanned the directories and removed these ephemeral snapshots when that marker file was found.
Since we are using SnapshotManager from now on (or at least we try to drift towards that), it is too soon to involve SnapshotManager at this very early stages of a node boot sequence. The stuff it needs to scan directories for snapshots (iterating over all Keyspaces and so on) is just not there yet.
In this new implementation, we indeed get rid of ephemerals too, but it is done a little bit later (still part of the booting sequence), just after all is initialised properly for SnapshotManager. Removal of ephemerals is done as one-time job and SnapshotManager is started in such a way that periodic checking for expired snapshots is not in effect yet. We are resuming this later in the booting sequence which will be obvious by a respective reader of the patch.
Also, this patch also introduces an interesting problem on upgrade path; imagine this scenario.
# I am running 4.0 where an ephemeral snapshot is taken and it is stored on disk with ephemeral marker file
# this node on 4.0 is turned off, so ephemeral snapshot still lives on disk because it is meant to be removed on next startup
# I upgrade this node to 4.1 (or whatever version this change will be in), but now we are expecting to have SnapshotManifest
# Since there is a manifest file from times when ephemeral flag was not introduced yet and marker file is not there anymore, by creating SnapshotManifest, it will be evaluated as that snapshot is _not_ ephemeral
# Since it is not ephemeral anymore, we just "promoted" a snapshot from ephemeral to normal one which will be never removed
I think that this problem has to be addressed by writing ephemeral marker files, as it is done now, in paralel with introducing this flag into SnapshotManifest so once a user is on 4.0 and he wants to upgrade to 4.1, ephemeral marker files will be still taken into account and removed on the startup, it is just that any new ephemeral snapshot will not have this marker file and flag in the manifest will b persisted instead of that.
In the next version, like 4.2, we will get rid of marker file logic completely because 4.1 will create all ephemeral snapshots with a flag in the manifest only.
> Remove ephemeral snapshot marker file and introduce a flag to SnapshotManifest
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-16911
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16911
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
>
> While creating an ephemeral snapshot, a marker file is created at disk in a respective snapshot directory. This is not necessary anymore as we have introduced SnapshotManifest in CASSANDRA-16789 so we can move this flag there. By putting the information if a snapshot is ephemeral or not into SnapshotManifest, we simplify and "clean up" the snapshotting process and related codebase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org