You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Akshit Jain <ja...@gmail.com> on 2019/10/30 12:49:24 UTC
Cassandra on Kubernetes
Hi everyone,
Is there anyone who is running Cassandra on K8s clusters. It would be great
if you can share your experience , the operator you are using and the
overall stability of stateful sets in Kubernetes
-Akshit
Re: Cassandra on Kubernetes
Posted by John Sanda <jo...@gmail.com>.
One of the problems I have experienced in the past has more to do with Java
than Cassandra in particular, and that is the JVM ignoring cgroups. With
Cassandra in particular I would often see memory usage go higher than what
was desired. This would lead to pods getting oom killed. This was fixed in
Java 10 though, and I believe even back ported to Java 8.
I think another issue is the lack of options for backup/restore.
I worked with Cassandra in Kubernetes pre-StatefulSets. That was a bit
rough :)
Local volumes were promoted to GA in Kubernetes 1.14. That is certainly a
good thing for stateful applications like Cassandra.
It is also important to have a sufficiently large value for the termination
grace period on pods to allow drain operation to complete, assuming you
perform a drain on shutdown.
On Wed, Oct 30, 2019 at 1:46 PM Akshit Jain <ja...@gmail.com> wrote:
> Hi Jean
> Thanks for replying. I had seen CassKop and the amount functionality it
> provides is quite awesome as compared to other operators.
>
> I would like to know how stable is kubernetes for stateful/database
> applications right now?
>
> I haven't read/heard any major production stateful application running on
> k8s.
>
>
> -Akshit
>
>
>
>
> On Wed, 30 Oct, 2019, 8:12 PM Jean-Armel Luce, <ja...@gmail.com> wrote:
>
>> Hi,
>>
>> We are currently developping CassKop, a Cassandra operator for K8S.
>> This operator is developped in Go, based on the operator-sdk framework.
>>
>> At this time of the project, the goal is to deploy a Cassandra cluster in
>> 1 Kubernetes datacenter, but this will change in next versions to deal with
>> Kubernetes in multi-datacenters.
>>
>> The following features are already supported by CassKop:
>> - Deployment of a C* cluster (rack or AZ aware)
>> - Scaling up the cluster (with cleanup)
>> - Scaling down the cluster (with decommission prior to Kubernetes scale
>> down)
>> - Pods operations (removenode, upgradesstable, cleanup, rebuild..)
>> - Adding a Cassandra DC
>> - Removing a Cassandra DC
>> - Setting and modifying configuration files
>> - Setting and modifying configuration parameters
>> - Update of the Cassandra docker image
>> - Rolling update of a Cassandra cluster
>> - Update of Cassandra version (including upgradesstable in case of major
>> upgrade)
>> - Update of JVM
>> - Update of configuration
>> - Stopping a Kubernetes node for maintenance
>> - Process a remove node (and create new Cassandra node on another
>> Kubernetes node)
>> - Process a replace address (of the old Cassandra node on another
>> Kubernetes node)
>> - Manage operations on pods through CassKop plugin (cleanup, rebuild,
>> upgradesstable, removenode..)
>> - Monitoring (using Instaclustr Prometheus exporter to Prometheus/Grafana)
>> - Pause/Restart & rolling restart operations through CassKoP plugin.
>>
>> We use also Cassandra reaper for scheduling repair sessions.
>>
>>
>> If you would like more informations about this operator, you may have a
>> look here : https://github.com/Orange-OpenSource/cassandra-k8s-operator
>>
>> Please, feel free to download it and try it. We would be more than happy
>> to receive your feedback
>>
>>
>> If you have any question about this operator, feel free to contact us via
>> our mailing-list: prj.casskop.support@list.orangeportails.net or on our
>> slack https://casskop.slack.com
>>
>> Note : this operator is still in alpha version and works only in a mono
>> region architecture for now. We are currently working hard for adding new
>> features in order to run it in multi-regions architecture.
>>
>>
>> Thanks.
>>
>>
>>
>> Le mer. 30 oct. 2019 à 13:56, Akshit Jain <ja...@gmail.com> a
>> écrit :
>>
>>> Hi everyone,
>>>
>>> Is there anyone who is running Cassandra on K8s clusters. It would be
>>> great if you can share your experience , the operator you are using and the
>>> overall stability of stateful sets in Kubernetes
>>>
>>> -Akshit
>>>
>>
--
- John
Re: Cassandra on Kubernetes
Posted by Jean-Armel Luce <ja...@gmail.com>.
Hi Jain,
Thanks for your comments about CassKop.
We began the development of Casskop at the beginning of 2018. At this time,
some K8S objects (i.e. statefulsets, operators, …) were still in beta
version and we discovered a few strange behaviours.
We upgraded to K8S 1.12 in mid-2018.
After this upgrade, we did not encounter any real problems.
However, we continue our tests; we have for example a cluster (16 nodes in
3 DC in the same region) which was deployed via CassKop and has been
working correctly for 3 months , and we will continue to observe it during
the next months before hopefully deploying a C* cluster via cassKop in
production.
In parallel, we do other tests of robustness and performances; right now,
we have no real issue to report about K8S for our use case.
I can’t say more right now.
Thanks for your question. I would be more than happy to read other answers.
Le mer. 30 oct. 2019 à 18:46, Akshit Jain <ja...@gmail.com> a écrit :
> Hi Jean
> Thanks for replying. I had seen CassKop and the amount functionality it
> provides is quite awesome as compared to other operators.
>
> I would like to know how stable is kubernetes for stateful/database
> applications right now?
>
> I haven't read/heard any major production stateful application running on
> k8s.
>
>
> -Akshit
>
>
>
>
> On Wed, 30 Oct, 2019, 8:12 PM Jean-Armel Luce, <ja...@gmail.com> wrote:
>
>> Hi,
>>
>> We are currently developping CassKop, a Cassandra operator for K8S.
>> This operator is developped in Go, based on the operator-sdk framework.
>>
>> At this time of the project, the goal is to deploy a Cassandra cluster in
>> 1 Kubernetes datacenter, but this will change in next versions to deal with
>> Kubernetes in multi-datacenters.
>>
>> The following features are already supported by CassKop:
>> - Deployment of a C* cluster (rack or AZ aware)
>> - Scaling up the cluster (with cleanup)
>> - Scaling down the cluster (with decommission prior to Kubernetes scale
>> down)
>> - Pods operations (removenode, upgradesstable, cleanup, rebuild..)
>> - Adding a Cassandra DC
>> - Removing a Cassandra DC
>> - Setting and modifying configuration files
>> - Setting and modifying configuration parameters
>> - Update of the Cassandra docker image
>> - Rolling update of a Cassandra cluster
>> - Update of Cassandra version (including upgradesstable in case of major
>> upgrade)
>> - Update of JVM
>> - Update of configuration
>> - Stopping a Kubernetes node for maintenance
>> - Process a remove node (and create new Cassandra node on another
>> Kubernetes node)
>> - Process a replace address (of the old Cassandra node on another
>> Kubernetes node)
>> - Manage operations on pods through CassKop plugin (cleanup, rebuild,
>> upgradesstable, removenode..)
>> - Monitoring (using Instaclustr Prometheus exporter to Prometheus/Grafana)
>> - Pause/Restart & rolling restart operations through CassKoP plugin.
>>
>> We use also Cassandra reaper for scheduling repair sessions.
>>
>>
>> If you would like more informations about this operator, you may have a
>> look here : https://github.com/Orange-OpenSource/cassandra-k8s-operator
>>
>> Please, feel free to download it and try it. We would be more than happy
>> to receive your feedback
>>
>>
>> If you have any question about this operator, feel free to contact us via
>> our mailing-list: prj.casskop.support@list.orangeportails.net or on our
>> slack https://casskop.slack.com
>>
>> Note : this operator is still in alpha version and works only in a mono
>> region architecture for now. We are currently working hard for adding new
>> features in order to run it in multi-regions architecture.
>>
>>
>> Thanks.
>>
>>
>>
>> Le mer. 30 oct. 2019 à 13:56, Akshit Jain <ja...@gmail.com> a
>> écrit :
>>
>>> Hi everyone,
>>>
>>> Is there anyone who is running Cassandra on K8s clusters. It would be
>>> great if you can share your experience , the operator you are using and the
>>> overall stability of stateful sets in Kubernetes
>>>
>>> -Akshit
>>>
>>
Re: Cassandra on Kubernetes
Posted by Akshit Jain <ja...@gmail.com>.
Hi Jean
Thanks for replying. I had seen CassKop and the amount functionality it
provides is quite awesome as compared to other operators.
I would like to know how stable is kubernetes for stateful/database
applications right now?
I haven't read/heard any major production stateful application running on
k8s.
-Akshit
On Wed, 30 Oct, 2019, 8:12 PM Jean-Armel Luce, <ja...@gmail.com> wrote:
> Hi,
>
> We are currently developping CassKop, a Cassandra operator for K8S.
> This operator is developped in Go, based on the operator-sdk framework.
>
> At this time of the project, the goal is to deploy a Cassandra cluster in
> 1 Kubernetes datacenter, but this will change in next versions to deal with
> Kubernetes in multi-datacenters.
>
> The following features are already supported by CassKop:
> - Deployment of a C* cluster (rack or AZ aware)
> - Scaling up the cluster (with cleanup)
> - Scaling down the cluster (with decommission prior to Kubernetes scale
> down)
> - Pods operations (removenode, upgradesstable, cleanup, rebuild..)
> - Adding a Cassandra DC
> - Removing a Cassandra DC
> - Setting and modifying configuration files
> - Setting and modifying configuration parameters
> - Update of the Cassandra docker image
> - Rolling update of a Cassandra cluster
> - Update of Cassandra version (including upgradesstable in case of major
> upgrade)
> - Update of JVM
> - Update of configuration
> - Stopping a Kubernetes node for maintenance
> - Process a remove node (and create new Cassandra node on another
> Kubernetes node)
> - Process a replace address (of the old Cassandra node on another
> Kubernetes node)
> - Manage operations on pods through CassKop plugin (cleanup, rebuild,
> upgradesstable, removenode..)
> - Monitoring (using Instaclustr Prometheus exporter to Prometheus/Grafana)
> - Pause/Restart & rolling restart operations through CassKoP plugin.
>
> We use also Cassandra reaper for scheduling repair sessions.
>
>
> If you would like more informations about this operator, you may have a
> look here : https://github.com/Orange-OpenSource/cassandra-k8s-operator
>
> Please, feel free to download it and try it. We would be more than happy
> to receive your feedback
>
>
> If you have any question about this operator, feel free to contact us via
> our mailing-list: prj.casskop.support@list.orangeportails.net or on our
> slack https://casskop.slack.com
>
> Note : this operator is still in alpha version and works only in a mono
> region architecture for now. We are currently working hard for adding new
> features in order to run it in multi-regions architecture.
>
>
> Thanks.
>
>
>
> Le mer. 30 oct. 2019 à 13:56, Akshit Jain <ja...@gmail.com> a
> écrit :
>
>> Hi everyone,
>>
>> Is there anyone who is running Cassandra on K8s clusters. It would be
>> great if you can share your experience , the operator you are using and the
>> overall stability of stateful sets in Kubernetes
>>
>> -Akshit
>>
>
Re: Cassandra on Kubernetes
Posted by Jean-Armel Luce <ja...@gmail.com>.
Hi,
We are currently developping CassKop, a Cassandra operator for K8S.
This operator is developped in Go, based on the operator-sdk framework.
At this time of the project, the goal is to deploy a Cassandra cluster in 1
Kubernetes datacenter, but this will change in next versions to deal with
Kubernetes in multi-datacenters.
The following features are already supported by CassKop:
- Deployment of a C* cluster (rack or AZ aware)
- Scaling up the cluster (with cleanup)
- Scaling down the cluster (with decommission prior to Kubernetes scale
down)
- Pods operations (removenode, upgradesstable, cleanup, rebuild..)
- Adding a Cassandra DC
- Removing a Cassandra DC
- Setting and modifying configuration files
- Setting and modifying configuration parameters
- Update of the Cassandra docker image
- Rolling update of a Cassandra cluster
- Update of Cassandra version (including upgradesstable in case of major
upgrade)
- Update of JVM
- Update of configuration
- Stopping a Kubernetes node for maintenance
- Process a remove node (and create new Cassandra node on another
Kubernetes node)
- Process a replace address (of the old Cassandra node on another
Kubernetes node)
- Manage operations on pods through CassKop plugin (cleanup, rebuild,
upgradesstable, removenode..)
- Monitoring (using Instaclustr Prometheus exporter to Prometheus/Grafana)
- Pause/Restart & rolling restart operations through CassKoP plugin.
We use also Cassandra reaper for scheduling repair sessions.
If you would like more informations about this operator, you may have a
look here : https://github.com/Orange-OpenSource/cassandra-k8s-operator
Please, feel free to download it and try it. We would be more than happy to
receive your feedback
If you have any question about this operator, feel free to contact us via
our mailing-list: prj.casskop.support@list.orangeportails.net or on our
slack https://casskop.slack.com
Note : this operator is still in alpha version and works only in a mono
region architecture for now. We are currently working hard for adding new
features in order to run it in multi-regions architecture.
Thanks.
Le mer. 30 oct. 2019 à 13:56, Akshit Jain <ja...@gmail.com> a écrit :
> Hi everyone,
>
> Is there anyone who is running Cassandra on K8s clusters. It would be
> great if you can share your experience , the operator you are using and the
> overall stability of stateful sets in Kubernetes
>
> -Akshit
>