You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jack Krupansky <ja...@gmail.com> on 2016/04/14 16:13:47 UTC
Experience with Kubernetes
Does anybody here have any experience, positive or negative, with deploying
Cassandra (or DSE) clusters using Kubernetes? I don't have any immediate
need (or experience), but I am curious about the pros and cons.
There is an example here:
https://github.com/kubernetes/kubernetes/tree/master/examples/cassandra
Is there a better approach to deploying a Cassandra/DSE cluster than
Kubernetes?
Thanks.
-- Jack Krupansky
Re: Experience with Kubernetes
Posted by Joe Stein <cr...@gmail.com>.
You can use Mesos https://github.com/elodina/datastax-enterprise-mesos
~ Joestein
On Apr 14, 2016 10:13 AM, "Jack Krupansky" <ja...@gmail.com> wrote:
> Does anybody here have any experience, positive or negative, with
> deploying Cassandra (or DSE) clusters using Kubernetes? I don't have any
> immediate need (or experience), but I am curious about the pros and cons.
>
> There is an example here:
> https://github.com/kubernetes/kubernetes/tree/master/examples/cassandra
>
> Is there a better approach to deploying a Cassandra/DSE cluster than
> Kubernetes?
>
> Thanks.
>
> -- Jack Krupansky
>
Re: Experience with Kubernetes
Posted by David Aronchick <ar...@gmail.com>.
We're working on that very documentation, but there's nothing about the
product that is inherently single datacenter. That is to say, if you
ALREADY wire together multiple data centers (via VPN, etc), then there's
zero about Kubernetes that wouldn't work great today (as folks are already
doing). What's your multi-DC solution today with VMs?
As far as using local, persistent storage, it's definitely not the core of
the containers design philosophy - the whole 'pets v. cattle' debate and
what not. However, if you'd like to use local storage, that's also trivial
to do in Kubernetes - It's called "hostPath" and it lets you mount a local
directory into your container.
http://kubernetes.io/docs/user-guide/volumes/#hostpath
However, you'd want to either limit your nodes to one container per node,
or you'll have IO contention. You can use that in Kubernetes using an
object called "DaemonSet" - http://kubernetes.io/docs/admin/daemons/
On Fri, Apr 15, 2016 at 3:31 PM Jack Krupansky <ja...@gmail.com>
wrote:
> It sounds as if Kubernetes is oriented towards a single data center, DC in
> Cassandra parlance, and maybe Ubernetes attempts to address that. If there
> is indeed some simple, obvious, way to trick Kubernetes into
> Cassandra-style multi-DC operation, it sure would be helpful if it was
> documented more obviously.
>
> Also, it seems like a lot of these cluster tools are oriented towards
> applications that in turn use networked services such as Cassandra for
> storage of the vast bulk of their data. IOW, they persist their real data
> in an external database such as Cassandra, and that any other persisted
> data or state is relatively inconsequential. How that model applies to a
> database server that works best with fairly large amounts of ultra-fast
> local data storage is not so obvious. Maybe that simply wasn't a design
> goal?
>
> -- Jack Krupansky
>
> On Fri, Apr 15, 2016 at 3:48 PM, David Aronchick <ar...@gmail.com>
> wrote:
>
>> Hi--
>>
>> It's trivial to do this in Kubernetes, even without Ubernetes. Please
>> feel free to send me a note and I'll walk you through it.
>>
>> Disclosure: I work on Google on Kubernetes.
>>
>> On Thu, Apr 14, 2016 at 9:10 AM Joe Stein <cr...@gmail.com> wrote:
>>
>>> You can do that with the Mesos scheduler
>>> https://github.com/elodina/datastax-enterprise-mesos and layout
>>> clusters and racks for datacenters based on attributes
>>> http://mesos.apache.org/documentation/latest/attributes-resources/
>>>
>>> ~ Joestein
>>> On Apr 14, 2016 12:05 PM, "Nate McCall" <na...@thelastpickle.com> wrote:
>>>
>>>>
>>>> Does anybody here have any experience, positive or negative, with
>>>>> deploying Cassandra (or DSE) clusters using Kubernetes? I don't have any
>>>>> immediate need (or experience), but I am curious about the pros and cons.
>>>>>
>>>>>
>>>>
>>>> The last time I played around with kubernetes+cassandra, you could not
>>>> specify node allocations across failure boundaries (AZs, Regions, etc).
>>>>
>>>> To me, that makes it not interesting outside of development or trivial
>>>> setups.
>>>>
>>>> It does look like they are getting farther along on "ubernetes" which
>>>> should fix this:
>>>>
>>>> https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md
>>>>
>>>>
>>>>
>>>> --
>>>> -----------------
>>>> Nate McCall
>>>> Austin, TX
>>>> @zznate
>>>>
>>>> Co-Founder & Sr. Technical Consultant
>>>> Apache Cassandra Consulting
>>>> http://www.thelastpickle.com
>>>>
>>>
>
Re: Experience with Kubernetes
Posted by Jack Krupansky <ja...@gmail.com>.
It sounds as if Kubernetes is oriented towards a single data center, DC in
Cassandra parlance, and maybe Ubernetes attempts to address that. If there
is indeed some simple, obvious, way to trick Kubernetes into
Cassandra-style multi-DC operation, it sure would be helpful if it was
documented more obviously.
Also, it seems like a lot of these cluster tools are oriented towards
applications that in turn use networked services such as Cassandra for
storage of the vast bulk of their data. IOW, they persist their real data
in an external database such as Cassandra, and that any other persisted
data or state is relatively inconsequential. How that model applies to a
database server that works best with fairly large amounts of ultra-fast
local data storage is not so obvious. Maybe that simply wasn't a design
goal?
-- Jack Krupansky
On Fri, Apr 15, 2016 at 3:48 PM, David Aronchick <ar...@gmail.com>
wrote:
> Hi--
>
> It's trivial to do this in Kubernetes, even without Ubernetes. Please feel
> free to send me a note and I'll walk you through it.
>
> Disclosure: I work on Google on Kubernetes.
>
> On Thu, Apr 14, 2016 at 9:10 AM Joe Stein <cr...@gmail.com> wrote:
>
>> You can do that with the Mesos scheduler
>> https://github.com/elodina/datastax-enterprise-mesos and layout clusters
>> and racks for datacenters based on attributes
>> http://mesos.apache.org/documentation/latest/attributes-resources/
>>
>> ~ Joestein
>> On Apr 14, 2016 12:05 PM, "Nate McCall" <na...@thelastpickle.com> wrote:
>>
>>>
>>> Does anybody here have any experience, positive or negative, with
>>>> deploying Cassandra (or DSE) clusters using Kubernetes? I don't have any
>>>> immediate need (or experience), but I am curious about the pros and cons.
>>>>
>>>>
>>>
>>> The last time I played around with kubernetes+cassandra, you could not
>>> specify node allocations across failure boundaries (AZs, Regions, etc).
>>>
>>> To me, that makes it not interesting outside of development or trivial
>>> setups.
>>>
>>> It does look like they are getting farther along on "ubernetes" which
>>> should fix this:
>>>
>>> https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md
>>>
>>>
>>>
>>> --
>>> -----------------
>>> Nate McCall
>>> Austin, TX
>>> @zznate
>>>
>>> Co-Founder & Sr. Technical Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>
Re: Experience with Kubernetes
Posted by David Aronchick <ar...@gmail.com>.
Hi--
It's trivial to do this in Kubernetes, even without Ubernetes. Please feel
free to send me a note and I'll walk you through it.
Disclosure: I work on Google on Kubernetes.
On Thu, Apr 14, 2016 at 9:10 AM Joe Stein <cr...@gmail.com> wrote:
> You can do that with the Mesos scheduler
> https://github.com/elodina/datastax-enterprise-mesos and layout clusters
> and racks for datacenters based on attributes
> http://mesos.apache.org/documentation/latest/attributes-resources/
>
> ~ Joestein
> On Apr 14, 2016 12:05 PM, "Nate McCall" <na...@thelastpickle.com> wrote:
>
>>
>> Does anybody here have any experience, positive or negative, with
>>> deploying Cassandra (or DSE) clusters using Kubernetes? I don't have any
>>> immediate need (or experience), but I am curious about the pros and cons.
>>>
>>>
>>
>> The last time I played around with kubernetes+cassandra, you could not
>> specify node allocations across failure boundaries (AZs, Regions, etc).
>>
>> To me, that makes it not interesting outside of development or trivial
>> setups.
>>
>> It does look like they are getting farther along on "ubernetes" which
>> should fix this:
>>
>> https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md
>>
>>
>>
>> --
>> -----------------
>> Nate McCall
>> Austin, TX
>> @zznate
>>
>> Co-Founder & Sr. Technical Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
Re: Experience with Kubernetes
Posted by Joe Stein <cr...@gmail.com>.
You can do that with the Mesos scheduler
https://github.com/elodina/datastax-enterprise-mesos and layout clusters
and racks for datacenters based on attributes
http://mesos.apache.org/documentation/latest/attributes-resources/
~ Joestein
On Apr 14, 2016 12:05 PM, "Nate McCall" <na...@thelastpickle.com> wrote:
>
> Does anybody here have any experience, positive or negative, with
>> deploying Cassandra (or DSE) clusters using Kubernetes? I don't have any
>> immediate need (or experience), but I am curious about the pros and cons.
>>
>>
>
> The last time I played around with kubernetes+cassandra, you could not
> specify node allocations across failure boundaries (AZs, Regions, etc).
>
> To me, that makes it not interesting outside of development or trivial
> setups.
>
> It does look like they are getting farther along on "ubernetes" which
> should fix this:
>
> https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
Re: Experience with Kubernetes
Posted by Nate McCall <na...@thelastpickle.com>.
> Does anybody here have any experience, positive or negative, with
> deploying Cassandra (or DSE) clusters using Kubernetes? I don't have any
> immediate need (or experience), but I am curious about the pros and cons.
>
>
The last time I played around with kubernetes+cassandra, you could not
specify node allocations across failure boundaries (AZs, Regions, etc).
To me, that makes it not interesting outside of development or trivial
setups.
It does look like they are getting farther along on "ubernetes" which
should fix this:
https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md
--
-----------------
Nate McCall
Austin, TX
@zznate
Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com