Snapshot creation after 3.4.14 -> 3.5.6 upgrade

[Thomas Cooper <to...@gmail.com>]

Hi,

I am working on upgrading a Kubernetes based 3.4.14 cluster to 3.5.6.

I are hitting the issue where the new snapshot checks prevent the newly
upgraded 3.5.6 cluster from starting. This usually happens in our test
set-up as the 3.4.14 cluster is not very old and has not hit the snapCount
limit yet. Typically only one node (presumable the leader) has a snapshot
file but the other do not.

I can disable the snapshot checks via the config value for the upgrade to
3.5.6, however we obviously need to re-enable the checks ASAP. The issue is
making sure all of the ZK nodes have snapshots so that the checks can be
re-enabled.

My question is: Is there a way to force the snapshot creation / sync from
the leader?

As part of our upgrade process we need to restart the servers soon after
the initial upgrade of the ZK container images to enable config changes. If
the snapshot creation / sync hasn't happened by this point then obviously
the checks will fail. If we can force a snapshot creation, after the
initial upgrade, we can re-enable the checks as part of the upgrade
process. If not, we will need to create some method to check if the
snapshots have been created and periodically inspect the ZK server file
system, re-enabling the checks once they all have snapshots.

Any pointers would be appreciated.

Thanks in advance,

Tom Cooper

Re: Snapshot creation after 3.4.14 -> 3.5.6 upgrade

[Michael Han <ha...@apache.org>]

>> My question is: Is there a way to force the snapshot creation / sync
from the leader?

3.5.6 will automatically create a clean snapshot as part of server start up
process. So a snapshot should be available after initial upgrade and there
is no need to force a snapshot creation.

On Mon, Dec 9, 2019 at 2:11 AM Thomas Cooper <to...@gmail.com> wrote:

> Hi,
>
> I am working on upgrading a Kubernetes based 3.4.14 cluster to 3.5.6.
>
> I are hitting the issue where the new snapshot checks prevent the newly
> upgraded 3.5.6 cluster from starting. This usually happens in our test
> set-up as the 3.4.14 cluster is not very old and has not hit the snapCount
> limit yet. Typically only one node (presumable the leader) has a snapshot
> file but the other do not.
>
> I can disable the snapshot checks via the config value for the upgrade to
> 3.5.6, however we obviously need to re-enable the checks ASAP. The issue is
> making sure all of the ZK nodes have snapshots so that the checks can be
> re-enabled.
>
> My question is: Is there a way to force the snapshot creation / sync from
> the leader?
>
> As part of our upgrade process we need to restart the servers soon after
> the initial upgrade of the ZK container images to enable config changes. If
> the snapshot creation / sync hasn't happened by this point then obviously
> the checks will fail. If we can force a snapshot creation, after the
> initial upgrade, we can re-enable the checks as part of the upgrade
> process. If not, we will need to create some method to check if the
> snapshots have been created and periodically inspect the ZK server file
> system, re-enabling the checks once they all have snapshots.
>
> Any pointers would be appreciated.
>
> Thanks in advance,
>
> Tom Cooper
>