Running and Managing Large Cassandra Clusters

[Gediminas Blazys <Ge...@microsoft.com.INVALID>]

Hello,

I wanted to seek out your opinion and experience.

Has anyone of you had a chance to run a Cassandra cluster of more than 350 nodes?
What are the major configuration considerations that you had to focus on? What number of vnodes did you use?
Once the cluster was up and running what would you have done differently?
Perhaps it would be more manageable to run multiple smaller clusters? Did you try this approach? What were the major challenges?

I don't know if questions like that are allowed here but I'm really interested in what other folks ran into while running massive operations.

Gediminas

Re: Running and Managing Large Cassandra Clusters

[Joshua McKenzie <jm...@apache.org>]

A few questions for you Tom if you have 30 seconds and care to disclose:

   1. What version of C*?
   2. What compaction strategy?
   3. What's core count allocated per C* node?
   4. Gossip give you any headaches / you have to be delicate there or does
   it behave itself?

Context: pmc/committer and I manage the OSS C* team at DataStax. We're
doing a lot of thinking about how to generally improve the operator
experience across the board for folks in the post 4.0 time frame, so data
like the above (where things are going well at scale and why) is super
useful to help feed into that effort.

Thanks!



On Wed, Oct 28, 2020 at 7:14 AM, Tom van der Woerdt <
tom.vanderwoerdt@booking.com.invalid> wrote:

> Does 360 count? :-)
>
> num_tokens is 16, works fine (had 256 on a 300 node cluster as well, not
> too many problems either). Roughly 2.5TB per node, running on-prem on
> reasonably stable hardware so replacements end up happening once a week at
> most, and there's no particular change needed in the automation. Scaling up
> or down takes a while, but it doesn't appear to be slower than any other
> cluster. Configuration wise it's no different than a 5-node cluster either.
> Pretty uneventful tbh.
>
> Tom van der Woerdt
> Senior Site Reliability Engineer
>
> Booking.com <http://booking.com/> BV
> Vijzelstraat Amsterdam Netherlands 1017HL
> [image: Booking.com] <https://www.booking.com/>
> Making it easier for everyone to experience the world since 1996
> 43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
> million reported listings
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>
>
> On Wed, Oct 28, 2020 at 8:58 AM Gediminas Blazys <Gediminas.Blazys@
> microsoft.com.invalid> wrote:
>
> Hello,
>
>
>
> I wanted to seek out your opinion and experience.
>
>
>
> Has anyone of you had a chance to run a Cassandra cluster of more than 350
> nodes?
>
> What are the major configuration considerations that you had to focus on?
> What number of vnodes did you use?
>
> Once the cluster was up and running what would you have done differently?
>
> Perhaps it would be more manageable to run multiple smaller clusters? Did
> you try this approach? What were the major challenges?
>
>
>
> I don’t know if questions like that are allowed here but I’m really
> interested in what other folks ran into while running massive operations.
>
>
>
> Gediminas
>
>

Re: Running and Managing Large Cassandra Clusters

[Tom van der Woerdt <to...@booking.com.INVALID>]

Does 360 count? :-)

num_tokens is 16, works fine (had 256 on a 300 node cluster as well, not
too many problems either). Roughly 2.5TB per node, running on-prem on
reasonably stable hardware so replacements end up happening once a week at
most, and there's no particular change needed in the automation. Scaling up
or down takes a while, but it doesn't appear to be slower than any other
cluster. Configuration wise it's no different than a 5-node cluster either.
Pretty uneventful tbh.

Tom van der Woerdt
Senior Site Reliability Engineer

Booking.com BV
Vijzelstraat Amsterdam Netherlands 1017HL
[image: Booking.com] <https://www.booking.com/>
Making it easier for everyone to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
million reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)


On Wed, Oct 28, 2020 at 8:58 AM Gediminas Blazys
<Ge...@microsoft.com.invalid> wrote:

> Hello,
>
>
>
> I wanted to seek out your opinion and experience.
>
>
>
> Has anyone of you had a chance to run a Cassandra cluster of more than 350
> nodes?
>
> What are the major configuration considerations that you had to focus on?
> What number of vnodes did you use?
>
> Once the cluster was up and running what would you have done differently?
>
> Perhaps it would be more manageable to run multiple smaller clusters? Did
> you try this approach? What were the major challenges?
>
>
>
> I don’t know if questions like that are allowed here but I’m really
> interested in what other folks ran into while running massive operations.
>
>
>
> Gediminas
>
>
>