Revive a downed node with a different IP address

[Antoine d'Otreppe <a....@aspyct.org>]

Hi all,

New to Cassandra, I'm trying to wrap my head around how dead nodes should be revived.

Specifically, we deployed our cluster in Kubernetes, which means that nodes that go down will lose their IP address. When restarted, it is possible that:

1. their IP address changes
2. their new IP address is that of another downed node.

I spent the last two days looking for, and reading, possible solutions online. However I could not find any recent or working solution (any link would be appreciated). I've seen plenty of hacks where people would define one k8s service per node, but that sounds like a burdensome and fragile solution.

My current understanding is that a node should be able to be revived and get its missing data from hinted handoff if it wasn't down longer than max_hint_handoff_windom. Or, if that window is exceeded, a repair would be needed. In any case, it's possible that the data is still available, and I'd like to avoid having to stream everything from zero from the other nodes.

I also looked into -Dcassandra.replace_address, but I feel like that would trigger a new token assignment, and again lots of streaming.

Finally there's one thing unclear to me as of yet (forgetting that dynamic IP address and kubernetes stuff): say I have several downed nodes, in the "DN" state. When one of those nodes is restarted, will it go through the "UJ" state? In other words, can I restart all downed nodes at once, or should I still respect the 2 minute rule?

And how would that work with dynamic IP addresses?

tl;dr: is there any updated documentation on how to revive nodes consistently when static IP addresses can't be assigned?

Best regards,
Antoine

Re: Revive a downed node with a different IP address

[Jeff Jirsa <jj...@gmail.com>]

On Wed, Feb 6, 2019 at 5:47 AM Antoine d'Otreppe <a....@aspyct.org>
wrote:

> Hi all,
>
> New to Cassandra, I'm trying to wrap my head around how dead nodes should
> be revived.
>
>
> Specifically, we deployed our cluster in Kubernetes, which means that
> nodes that go down will lose their IP address. When restarted, it is
> possible that:
>
> 1. their IP address changes
>

This in itself is not a problem, but


> 2. their new IP address is that of another downed node.
>

This ends up being a huge problem in cassandra with K8s. Since we use just
the bare IP as the key for some data structures, re-using the IP of another
down instance basically (incorrectly) removes it from the ring.


>
> I spent the last two days looking for, and reading, possible solutions
> online. However I could not find any recent or working solution (any link
> would be appreciated). I've seen plenty of hacks where people would define
> one k8s service per node, but that sounds like a burdensome and fragile
> solution.
>

What you may want to consider as a workaround is starting a pod and then
interrogating the assigned IP to see if it already exists / DOWN in the
cluster before you issue the start command for Cassandra itself.


>
> My current understanding is that a node should be able to be revived and
> get its missing data from hinted handoff if it wasn't down longer than
> max_hint_handoff_windom.
>
Or, if that window is exceeded, a repair would be needed. In any case, it's
> possible that the data is still available, and I'd like to avoid having to
> stream everything from zero from the other nodes.
>

This is correct. If you don't do deletes, then the max_hint_handoff_window
becomes MUCH less important.


>
> I also looked into -Dcassandra.replace_address, but I feel like that would
> trigger a new token assignment, and again lots of streaming.
>

It puts a new instance on top of the old, down instance. Strictly speaking,
you'd want to run repair BEFORE you start streaming or you violate
consistency, so you'd have to repair, then re-stream a whole instance of
data.


>
> Finally there's one thing unclear to me as of yet (forgetting that dynamic
> IP address and kubernetes stuff): say I have several downed nodes, in the
> "DN" state. When one of those nodes is restarted, will it go through the
> "UJ" state?
>

No, it'll go straight to UN


> In other words, can I restart all downed nodes at once, or should I still
> respect the 2 minute rule?
>

You can restart them all at once.


>
> And how would that work with dynamic IP addresses?
>

As mentioned before, you need a way to avoid having a restarting instance
take the IP of another instance already in the cluster.


>
>
> tl;dr: is there any updated documentation on how to revive nodes
> consistently when static IP addresses can't be assigned?
>


Probably not documented, no.


>
>
> Best regards,
> Antoine
>