You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by manish khandelwal <ma...@gmail.com> on 2020/07/06 17:01:01 UTC
Cassandra on K8S
I have started reading about how to deploy Cassandra with K8S. But as I
read more I feel there are a lot of challenges in running Cassandra on K8s.
Some of the challenges which I feel are
1. POD IPs identification - If the pods go down and when they come up their
IPs change, how is it handled as we are dependent on IPs of Cassandra nodes
for internode as well client server communication.
2. A K8S node can host a single pod. This is being done so that even if
the host goes down we have only one pod down case. With multiple pods on a
single host there is a risk of traffic failures as consistency might not be
achieved. But if we keep two pods of the same rack on a single host then
are we safe or is there any unknown risk?
3. Seed discovery? Again as an extension of point 1, since IPs can change,
how we can manage seeds.
4. Also I read a lot of use of Cassandra operators for maintaining a
Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a
robot (automated admin) which works and acts like a norma admin will work.
I want to understand that how important is Cassandra operator and what if
we go on to production without one?
Regards
Manish
Re: Cassandra on K8S
Posted by manish khandelwal <ma...@gmail.com>.
I am asking this because the only case where I see IP swap can occur is
when two Cassandra nodes are running on the same K8S host node. I am
evaluating how safe it is two run two Cassandra nodes on a single K8S host
node.
*Totally Agree that swap is not the right word but one node still taking
other nodes IP*
Regards
Manish
On Tue, Aug 4, 2020 at 10:07 AM manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> But again if
> Some Cassandra node (pod) with particular IP X is down,
> Second Cassandra node (pod) tries to take the IP X of first Cassandra node,
> Second Cassandra node should fail to join the cluster as the Cassandra
> cluster will complain that IP X is already occupied.
> In that sense actual swap of IPs can occur between two nodes and data
> issues should not occur.
>
> I am asking this because the only case where I see IP swap can occur is
> when two Cassandra nodes are running on the same K8S host node. I am
> evaluating how safe it is two run two Cassandra nodes on a single K8S host
> node.
>
> Regards
> Manish
>
>
>
> On Tue, Aug 4, 2020 at 9:01 AM Christopher Bradford <br...@gmail.com>
> wrote:
>
>> In *most* k8s environments each Kubernetes worker receives its own
>> dedicated CIDR range from the cluster’s CIDR space for allocating pod IP
>> addresses. The issue described can occur when a k8s worker goes down then
>> comes back up and the pods are rescheduled where either pod starts up with
>> another pods previously used IP.
>>
>> They don’t necessarily have to swap 1:1 (ie one pod could use the other’s
>> previous address while that pod receives a new address). Additionally it’s
>> not a race condition of which container starts first. The k8s scheduler and
>> kubelet daemon assign IPs to pods.
>>
>> On Mon, Aug 3, 2020 at 11:14 PM manish khandelwal <
>> manishkhandelwal03@gmail.com> wrote:
>>
>>> I have started reading about how to deploy Cassandra with K8S. But as I
>>>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>>>> Some of the challenges which I feel are
>>>>
>>>> 1. POD IPs identification - If the pods go down and when they come up
>>>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>>>> nodes for internode as well client server communication.
>>>>
>>>
>>> Strictly safe to change an IP to an IP that is unused.
>>> Strictly unsafe to use an IP that's already in the cluster (so if two
>>> pods go down, having the first pod that comes up grab the IP of the second
>>> pod is strictly dangerous and will violate consistency and maybe lose
>>> data).
>>>
>>> *For point 2 (Strictly unsafe to use an IP) , if the first pod grabs the
>>> IP of the second node then it should not be able to join the cluster.*
>>> *So can IPs still be swapped? *
>>> *When and how this IP swap can occur?*
>>>
>>> Regards
>>> Manish
>>>
>>> On Mon, Jul 6, 2020 at 10:40 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal <
>>>> manishkhandelwal03@gmail.com> wrote:
>>>>
>>>>> I have started reading about how to deploy Cassandra with K8S. But as
>>>>> I read more I feel there are a lot of challenges in running Cassandra on
>>>>> K8s. Some of the challenges which I feel are
>>>>>
>>>>> 1. POD IPs identification - If the pods go down and when they come up
>>>>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>>>>> nodes for internode as well client server communication.
>>>>>
>>>>
>>>> Strictly safe to change an IP to an IP that is unused.
>>>> Strictly unsafe to use an IP that's already in the cluster (so if two
>>>> pods go down, having the first pod that comes up grab the IP of the second
>>>> pod is strictly dangerous and will violate consistency and maybe lose
>>>> data).
>>>>
>>>>
>>>>>
>>>>> 2. A K8S node can host a single pod. This is being done so that even
>>>>> if the host goes down we have only one pod down case. With multiple pods on
>>>>> a single host there is a risk of traffic failures as consistency might not
>>>>> be achieved. But if we keep two pods of the same rack on a single host then
>>>>> are we safe or is there any unknown risk?
>>>>>
>>>>
>>>> This sounds like you're asking if rack aware snitches protect you from
>>>> concurrent pods going down. Make sure you're using a rack aware snitch.
>>>>
>>>>
>>>>>
>>>>> 3. Seed discovery? Again as an extension of point 1, since IPs can
>>>>> change, how we can manage seeds.
>>>>>
>>>>
>>>> Either use DNS instead of static IPs, or use a seed provider that
>>>> handles IPs changing.
>>>>
>>>>
>>>>>
>>>>> 4. Also I read a lot of use of Cassandra operators for maintaining a
>>>>> Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a
>>>>> robot (automated admin) which works and acts like a norma admin will work.
>>>>> I want to understand that how important is Cassandra operator and what if
>>>>> we go on to production without one?
>>>>>
>>>>> Regards
>>>>> Manish
>>>>>
>>>> --
>>
>> Christopher Bradford
>>
>>
Re: Cassandra on K8S
Posted by manish khandelwal <ma...@gmail.com>.
But again if
Some Cassandra node (pod) with particular IP X is down,
Second Cassandra node (pod) tries to take the IP X of first Cassandra node,
Second Cassandra node should fail to join the cluster as the Cassandra
cluster will complain that IP X is already occupied.
In that sense actual swap of IPs can occur between two nodes and data
issues should not occur.
I am asking this because the only case where I see IP swap can occur is
when two Cassandra nodes are running on the same K8S host node. I am
evaluating how safe it is two run two Cassandra nodes on a single K8S host
node.
Regards
Manish
On Tue, Aug 4, 2020 at 9:01 AM Christopher Bradford <br...@gmail.com>
wrote:
> In *most* k8s environments each Kubernetes worker receives its own
> dedicated CIDR range from the cluster’s CIDR space for allocating pod IP
> addresses. The issue described can occur when a k8s worker goes down then
> comes back up and the pods are rescheduled where either pod starts up with
> another pods previously used IP.
>
> They don’t necessarily have to swap 1:1 (ie one pod could use the other’s
> previous address while that pod receives a new address). Additionally it’s
> not a race condition of which container starts first. The k8s scheduler and
> kubelet daemon assign IPs to pods.
>
> On Mon, Aug 3, 2020 at 11:14 PM manish khandelwal <
> manishkhandelwal03@gmail.com> wrote:
>
>> I have started reading about how to deploy Cassandra with K8S. But as I
>>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>>> Some of the challenges which I feel are
>>>
>>> 1. POD IPs identification - If the pods go down and when they come up
>>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>>> nodes for internode as well client server communication.
>>>
>>
>> Strictly safe to change an IP to an IP that is unused.
>> Strictly unsafe to use an IP that's already in the cluster (so if two
>> pods go down, having the first pod that comes up grab the IP of the second
>> pod is strictly dangerous and will violate consistency and maybe lose
>> data).
>>
>> *For point 2 (Strictly unsafe to use an IP) , if the first pod grabs the
>> IP of the second node then it should not be able to join the cluster.*
>> *So can IPs still be swapped? *
>> *When and how this IP swap can occur?*
>>
>> Regards
>> Manish
>>
>> On Mon, Jul 6, 2020 at 10:40 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>>
>>>
>>> On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal <
>>> manishkhandelwal03@gmail.com> wrote:
>>>
>>>> I have started reading about how to deploy Cassandra with K8S. But as I
>>>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>>>> Some of the challenges which I feel are
>>>>
>>>> 1. POD IPs identification - If the pods go down and when they come up
>>>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>>>> nodes for internode as well client server communication.
>>>>
>>>
>>> Strictly safe to change an IP to an IP that is unused.
>>> Strictly unsafe to use an IP that's already in the cluster (so if two
>>> pods go down, having the first pod that comes up grab the IP of the second
>>> pod is strictly dangerous and will violate consistency and maybe lose
>>> data).
>>>
>>>
>>>>
>>>> 2. A K8S node can host a single pod. This is being done so that even
>>>> if the host goes down we have only one pod down case. With multiple pods on
>>>> a single host there is a risk of traffic failures as consistency might not
>>>> be achieved. But if we keep two pods of the same rack on a single host then
>>>> are we safe or is there any unknown risk?
>>>>
>>>
>>> This sounds like you're asking if rack aware snitches protect you from
>>> concurrent pods going down. Make sure you're using a rack aware snitch.
>>>
>>>
>>>>
>>>> 3. Seed discovery? Again as an extension of point 1, since IPs can
>>>> change, how we can manage seeds.
>>>>
>>>
>>> Either use DNS instead of static IPs, or use a seed provider that
>>> handles IPs changing.
>>>
>>>
>>>>
>>>> 4. Also I read a lot of use of Cassandra operators for maintaining a
>>>> Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a
>>>> robot (automated admin) which works and acts like a norma admin will work.
>>>> I want to understand that how important is Cassandra operator and what if
>>>> we go on to production without one?
>>>>
>>>> Regards
>>>> Manish
>>>>
>>> --
>
> Christopher Bradford
>
>
Re: Cassandra on K8S
Posted by Christopher Bradford <br...@gmail.com>.
In *most* k8s environments each Kubernetes worker receives its own
dedicated CIDR range from the cluster’s CIDR space for allocating pod IP
addresses. The issue described can occur when a k8s worker goes down then
comes back up and the pods are rescheduled where either pod starts up with
another pods previously used IP.
They don’t necessarily have to swap 1:1 (ie one pod could use the other’s
previous address while that pod receives a new address). Additionally it’s
not a race condition of which container starts first. The k8s scheduler and
kubelet daemon assign IPs to pods.
On Mon, Aug 3, 2020 at 11:14 PM manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> I have started reading about how to deploy Cassandra with K8S. But as I
>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>> Some of the challenges which I feel are
>>
>> 1. POD IPs identification - If the pods go down and when they come up
>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>> nodes for internode as well client server communication.
>>
>
> Strictly safe to change an IP to an IP that is unused.
> Strictly unsafe to use an IP that's already in the cluster (so if two pods
> go down, having the first pod that comes up grab the IP of the second pod
> is strictly dangerous and will violate consistency and maybe lose data).
>
> *For point 2 (Strictly unsafe to use an IP) , if the first pod grabs the
> IP of the second node then it should not be able to join the cluster.*
> *So can IPs still be swapped? *
> *When and how this IP swap can occur?*
>
> Regards
> Manish
>
> On Mon, Jul 6, 2020 at 10:40 PM Jeff Jirsa <jj...@gmail.com> wrote:
>
>>
>>
>> On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal <
>> manishkhandelwal03@gmail.com> wrote:
>>
>>> I have started reading about how to deploy Cassandra with K8S. But as I
>>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>>> Some of the challenges which I feel are
>>>
>>> 1. POD IPs identification - If the pods go down and when they come up
>>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>>> nodes for internode as well client server communication.
>>>
>>
>> Strictly safe to change an IP to an IP that is unused.
>> Strictly unsafe to use an IP that's already in the cluster (so if two
>> pods go down, having the first pod that comes up grab the IP of the second
>> pod is strictly dangerous and will violate consistency and maybe lose
>> data).
>>
>>
>>>
>>> 2. A K8S node can host a single pod. This is being done so that even if
>>> the host goes down we have only one pod down case. With multiple pods on a
>>> single host there is a risk of traffic failures as consistency might not be
>>> achieved. But if we keep two pods of the same rack on a single host then
>>> are we safe or is there any unknown risk?
>>>
>>
>> This sounds like you're asking if rack aware snitches protect you from
>> concurrent pods going down. Make sure you're using a rack aware snitch.
>>
>>
>>>
>>> 3. Seed discovery? Again as an extension of point 1, since IPs can
>>> change, how we can manage seeds.
>>>
>>
>> Either use DNS instead of static IPs, or use a seed provider that handles
>> IPs changing.
>>
>>
>>>
>>> 4. Also I read a lot of use of Cassandra operators for maintaining a
>>> Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a
>>> robot (automated admin) which works and acts like a norma admin will work.
>>> I want to understand that how important is Cassandra operator and what if
>>> we go on to production without one?
>>>
>>> Regards
>>> Manish
>>>
>> --
Christopher Bradford
Re: Cassandra on K8S
Posted by manish khandelwal <ma...@gmail.com>.
>
> I have started reading about how to deploy Cassandra with K8S. But as I
> read more I feel there are a lot of challenges in running Cassandra on K8s.
> Some of the challenges which I feel are
>
> 1. POD IPs identification - If the pods go down and when they come up
> their IPs change, how is it handled as we are dependent on IPs of Cassandra
> nodes for internode as well client server communication.
>
Strictly safe to change an IP to an IP that is unused.
Strictly unsafe to use an IP that's already in the cluster (so if two pods
go down, having the first pod that comes up grab the IP of the second pod
is strictly dangerous and will violate consistency and maybe lose data).
*For point 2 (Strictly unsafe to use an IP) , if the first pod grabs the IP
of the second node then it should not be able to join the cluster.*
*So can IPs still be swapped? *
*When and how this IP swap can occur?*
Regards
Manish
On Mon, Jul 6, 2020 at 10:40 PM Jeff Jirsa <jj...@gmail.com> wrote:
>
>
> On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal <
> manishkhandelwal03@gmail.com> wrote:
>
>> I have started reading about how to deploy Cassandra with K8S. But as I
>> read more I feel there are a lot of challenges in running Cassandra on K8s.
>> Some of the challenges which I feel are
>>
>> 1. POD IPs identification - If the pods go down and when they come up
>> their IPs change, how is it handled as we are dependent on IPs of Cassandra
>> nodes for internode as well client server communication.
>>
>
> Strictly safe to change an IP to an IP that is unused.
> Strictly unsafe to use an IP that's already in the cluster (so if two pods
> go down, having the first pod that comes up grab the IP of the second pod
> is strictly dangerous and will violate consistency and maybe lose data).
>
>
>>
>> 2. A K8S node can host a single pod. This is being done so that even if
>> the host goes down we have only one pod down case. With multiple pods on a
>> single host there is a risk of traffic failures as consistency might not be
>> achieved. But if we keep two pods of the same rack on a single host then
>> are we safe or is there any unknown risk?
>>
>
> This sounds like you're asking if rack aware snitches protect you from
> concurrent pods going down. Make sure you're using a rack aware snitch.
>
>
>>
>> 3. Seed discovery? Again as an extension of point 1, since IPs can
>> change, how we can manage seeds.
>>
>
> Either use DNS instead of static IPs, or use a seed provider that handles
> IPs changing.
>
>
>>
>> 4. Also I read a lot of use of Cassandra operators for maintaining a
>> Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a
>> robot (automated admin) which works and acts like a norma admin will work.
>> I want to understand that how important is Cassandra operator and what if
>> we go on to production without one?
>>
>> Regards
>> Manish
>>
>
Re: Cassandra on K8S
Posted by Jeff Jirsa <jj...@gmail.com>.
On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> I have started reading about how to deploy Cassandra with K8S. But as I
> read more I feel there are a lot of challenges in running Cassandra on K8s.
> Some of the challenges which I feel are
>
> 1. POD IPs identification - If the pods go down and when they come up
> their IPs change, how is it handled as we are dependent on IPs of Cassandra
> nodes for internode as well client server communication.
>
Strictly safe to change an IP to an IP that is unused.
Strictly unsafe to use an IP that's already in the cluster (so if two pods
go down, having the first pod that comes up grab the IP of the second pod
is strictly dangerous and will violate consistency and maybe lose data).
>
> 2. A K8S node can host a single pod. This is being done so that even if
> the host goes down we have only one pod down case. With multiple pods on a
> single host there is a risk of traffic failures as consistency might not be
> achieved. But if we keep two pods of the same rack on a single host then
> are we safe or is there any unknown risk?
>
This sounds like you're asking if rack aware snitches protect you from
concurrent pods going down. Make sure you're using a rack aware snitch.
>
> 3. Seed discovery? Again as an extension of point 1, since IPs can change,
> how we can manage seeds.
>
Either use DNS instead of static IPs, or use a seed provider that handles
IPs changing.
>
> 4. Also I read a lot of use of Cassandra operators for maintaining a
> Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a
> robot (automated admin) which works and acts like a norma admin will work.
> I want to understand that how important is Cassandra operator and what if
> we go on to production without one?
>
> Regards
> Manish
>