You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jens Rantil <je...@tink.se> on 2016/09/11 21:55:14 UTC
Re: Cassandra and Kubernetes and scaling
Hi Aiman,
I noticed you never got any reply. This might be of interest:
http://blog.kubernetes.io/2016/07/thousand-instances-of-cassandra-using-kubernetes-pet-set.html
Cheers,
Jens
On Tuesday, May 24, 2016, Aiman Parvaiz <ai...@flipagram.com> wrote:
> Looking forward to hearing from the community about this.
>
> Sent from my iPhone
>
> > On May 24, 2016, at 10:19 AM, Mike Wojcikiewicz <mike@withkash.com
> <javascript:;>> wrote:
> >
> > I saw a thread from April 2016 talking about Cassandra and Kubernetes,
> and have a few follow up questions. It seems that especially after v1.2 of
> Kubernetes, and the upcoming 1.3 features, this would be a very viable
> option of running Cassandra on.
> >
> > My questions pertain to HostIds and Scaling Up/Down, and are related:
> >
> > 1. If a container's host dies and is then brought up on another host,
> can you start up with the same PersistentVolume as the original container
> had? Which begs the question would the new container get a new HostId,
> implying it would need to bootstrap into the environment? If it's a
> bootstrap, does the old one get deco'd/assassinated?
> >
> > 2. Scaling up/down. Scaling up would be relatively easy, as it should
> just kick off Bootstrapping the node into the cluster, but what if you need
> to scale down? Would the Container get deco'd by the scaling down process?
> or just terminated, leaving you with potential missing replicas
> >
> > 3. Scaling up and increasing the RF of a particular keyspace, would
> there be a clean way to do this with the kubernetes tooling?
> >
> > In the end I'm wondering how much of the Kubernetes + Cassandra involves
> nodetool, and how much is just a Docker image where you need to manage that
> all yourself (painfully)
> >
> > --
> > --mike
>
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
Twitter <https://twitter.com/tink>
Re: Cassandra and Kubernetes and scaling
Posted by Jens Rantil <je...@tink.se>.
David,
Were you the one who wrote the article? I just finished reading it. It's
excellent! I'm also excited that running mutable infrastructure on
containers is maturing. I have a few specific questions you (or someone
else!) might be able to answer.
1. In the article you state
> We deployed 1,009 minion nodes to Google Compute Engine
<https://cloud.google.com/compute/> (GCE), spread across 4 zones, running a
custom version of the Kubernetes 1.3 beta.
Did you deploy a custom Kubernetes on GCE because 1.3 wasn't available? Or
was that because Pet Sets alpha feature was disabled on Google Cloud
Platform's hosted Kubernetes[1]?
[1] http://serverfault.com/q/802437/37237
2. The article stated
> Yes we deployed 1,000 pets, but one really did not want to join the party!
Do you have any speculation why this happened? By default Cassandra doesn't
allow concurrent nodes joining the cluster, but Pet Sets are added serially
by definition, right?
3. The article doesn't mention downscaling. Do you have any idea on how
that would/could be done? I consider myself a Kubernetes/container noob. It
there an equivalent of `readinessProbe` for shutting down containers? Or
would an external agent have to be deployed that orchestrates `nodetool
decommission`s an instance and then reduces the number of replicas by one
for the Pet Set?
4. For a smaller number of Cassandra nodes. Would you feel comfortable
running it on Kubernetes 1.3? ;)
Cheers,
Jens
On Monday, September 12, 2016, David Aronchick <ar...@gmail.com> wrote:
> Please let me know if I can help at all!
>
> On Sun, Sep 11, 2016 at 2:55 PM, Jens Rantil <jens.rantil@tink.se
> <javascript:_e(%7B%7D,'cvml','jens.rantil@tink.se');>> wrote:
>
>> Hi Aiman,
>>
>> I noticed you never got any reply. This might be of interest:
>> http://blog.kubernetes.io/2016/07/thousand-instances-of-cassandra-using-
>> kubernetes-pet-set.html
>>
>> Cheers,
>> Jens
>>
>> On Tuesday, May 24, 2016, Aiman Parvaiz <aiman@flipagram.com
>> <javascript:_e(%7B%7D,'cvml','aiman@flipagram.com');>> wrote:
>>
>>> Looking forward to hearing from the community about this.
>>>
>>> Sent from my iPhone
>>>
>>> > On May 24, 2016, at 10:19 AM, Mike Wojcikiewicz <mi...@withkash.com>
>>> wrote:
>>> >
>>> > I saw a thread from April 2016 talking about Cassandra and Kubernetes,
>>> and have a few follow up questions. It seems that especially after v1.2 of
>>> Kubernetes, and the upcoming 1.3 features, this would be a very viable
>>> option of running Cassandra on.
>>> >
>>> > My questions pertain to HostIds and Scaling Up/Down, and are related:
>>> >
>>> > 1. If a container's host dies and is then brought up on another host,
>>> can you start up with the same PersistentVolume as the original container
>>> had? Which begs the question would the new container get a new HostId,
>>> implying it would need to bootstrap into the environment? If it's a
>>> bootstrap, does the old one get deco'd/assassinated?
>>> >
>>> > 2. Scaling up/down. Scaling up would be relatively easy, as it should
>>> just kick off Bootstrapping the node into the cluster, but what if you need
>>> to scale down? Would the Container get deco'd by the scaling down process?
>>> or just terminated, leaving you with potential missing replicas
>>> >
>>> > 3. Scaling up and increasing the RF of a particular keyspace, would
>>> there be a clean way to do this with the kubernetes tooling?
>>> >
>>> > In the end I'm wondering how much of the Kubernetes + Cassandra
>>> involves nodetool, and how much is just a Docker image where you need to
>>> manage that all yourself (painfully)
>>> >
>>> > --
>>> > --mike
>>>
>>
>>
>> --
>> Jens Rantil
>> Backend engineer
>> Tink AB
>>
>> Email: jens.rantil@tink.se
>> <javascript:_e(%7B%7D,'cvml','jens.rantil@tink.se');>
>> Phone: +46 708 84 18 32
>> Web: www.tink.se
>>
>> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
>> <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
>> Twitter <https://twitter.com/tink>
>>
>>
>
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
Twitter <https://twitter.com/tink>
Re: Cassandra and Kubernetes and scaling
Posted by David Aronchick <ar...@gmail.com>.
Please let me know if I can help at all!
On Sun, Sep 11, 2016 at 2:55 PM, Jens Rantil <je...@tink.se> wrote:
> Hi Aiman,
>
> I noticed you never got any reply. This might be of interest: http://blog.
> kubernetes.io/2016/07/thousand-instances-of-cassandra-using-kubernetes-
> pet-set.html
>
> Cheers,
> Jens
>
> On Tuesday, May 24, 2016, Aiman Parvaiz <ai...@flipagram.com> wrote:
>
>> Looking forward to hearing from the community about this.
>>
>> Sent from my iPhone
>>
>> > On May 24, 2016, at 10:19 AM, Mike Wojcikiewicz <mi...@withkash.com>
>> wrote:
>> >
>> > I saw a thread from April 2016 talking about Cassandra and Kubernetes,
>> and have a few follow up questions. It seems that especially after v1.2 of
>> Kubernetes, and the upcoming 1.3 features, this would be a very viable
>> option of running Cassandra on.
>> >
>> > My questions pertain to HostIds and Scaling Up/Down, and are related:
>> >
>> > 1. If a container's host dies and is then brought up on another host,
>> can you start up with the same PersistentVolume as the original container
>> had? Which begs the question would the new container get a new HostId,
>> implying it would need to bootstrap into the environment? If it's a
>> bootstrap, does the old one get deco'd/assassinated?
>> >
>> > 2. Scaling up/down. Scaling up would be relatively easy, as it should
>> just kick off Bootstrapping the node into the cluster, but what if you need
>> to scale down? Would the Container get deco'd by the scaling down process?
>> or just terminated, leaving you with potential missing replicas
>> >
>> > 3. Scaling up and increasing the RF of a particular keyspace, would
>> there be a clean way to do this with the kubernetes tooling?
>> >
>> > In the end I'm wondering how much of the Kubernetes + Cassandra
>> involves nodetool, and how much is just a Docker image where you need to
>> manage that all yourself (painfully)
>> >
>> > --
>> > --mike
>>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
> Twitter <https://twitter.com/tink>
>
>