Do vnodes need more memory?

[Tom van den Berge <to...@gmail.com>]

I have two data centers, each with the same number of nodes, same hardware
(CPUs, memory), Cassandra version (2.1.6), replication factory, etc. The
only difference it that one data center uses vnodes, and the other doesn't.

The non-vnode DC works fine (and has been for a long time) under production
load: I'm seeing normal CPU and IO load and garbage collection figures. But
the vnode DC is struggling very hard under the same load. It has been set
up recently. The CPU load is very high, due to excessive garbage collection
(>50% of the time is spent collecting).

So it seems that Cassandra simply doesn't have enough memory. I'm trying to
understand if this can be caused by the use of vnodes? Is there an sensible
reason why vnodes would consume more memory than regular nodes? Or does any
of you have the same experience? If not, I might be barking up the wrong
tree here, and I would love to know it before upgrading my servers with
more memory.

Thanks,
Tom

Re: Do vnodes need more memory?

[Tom van den Berge <to...@gmail.com>]

On Thu, Sep 24, 2015 at 12:45 AM, Robert Coli <rc...@eventbrite.com> wrote:

> On Wed, Sep 23, 2015 at 7:09 AM, Tom van den Berge <
> tom.vandenberge@gmail.com> wrote:
>
>> So it seems that Cassandra simply doesn't have enough memory. I'm trying
>> to understand if this can be caused by the use of vnodes? Is there an
>> sensible reason why vnodes would consume more memory than regular nodes? Or
>> does any of you have the same experience? If not, I might be barking up the
>> wrong tree here, and I would love to know it before upgrading my servers
>> with more memory.
>>
>
> Yes, range ownership has a RAM/heap cost per-range-owned. This cost is
> paid during many, but not all, operations. Owning 256 ranges > Owning 1
> range.
>
> I have not had the same experience but am not at all surprised to hear
> that vnodes increase heap consumption for otherwise identical
> configurations. I am surprised to hear that it makes a significant
> difference in GC time, but you might have been close enough to heap
> saturation that vnodes tip you over.
>

That's apparently exactly what's going on. We've just increased the memory
from 8 to 16 GB, and all is fine now. This seems to confirm that using
vnodes indeed increase heap consumption significantly. I think it would be
great if this could be advertised in the documentation, as a warning. From
the current documentation, it seems that vnodes don't come at any cost.

What's also interesting is this: Before increasing the memory we have been
changing our code not to use our secondary indexes anymore. We still had a
number of those, and we were suspecting them to be the cause of the
increased heap consumption. It did not eliminate the problem, but it
definitely helped to bring the GC times dramatically. I already knew that
secondary indexes are best not to use, but it seems that using them in
combination with vnodes makes it far worse.


> As an aside, one is likely to win very little net win from vnodes if one's
> cluster is not now and will never be more than approximately 15 nodes.
>

That's a very interesting observation. Especially since vnodes are enabled
by default for some time now, and apparently it has a (heap) price. And my
guess is that a significant percentage of all clusters will never exceed 15
nodes.

Thx,
Tom

Re: Do vnodes need more memory?

[Robert Coli <rc...@eventbrite.com>]

On Wed, Sep 23, 2015 at 7:09 AM, Tom van den Berge <
tom.vandenberge@gmail.com> wrote:

> So it seems that Cassandra simply doesn't have enough memory. I'm trying
> to understand if this can be caused by the use of vnodes? Is there an
> sensible reason why vnodes would consume more memory than regular nodes? Or
> does any of you have the same experience? If not, I might be barking up the
> wrong tree here, and I would love to know it before upgrading my servers
> with more memory.
>

Yes, range ownership has a RAM/heap cost per-range-owned. This cost is paid
during many, but not all, operations. Owning 256 ranges > Owning 1 range.

I have not had the same experience but am not at all surprised to hear that
vnodes increase heap consumption for otherwise identical configurations. I
am surprised to hear that it makes a significant difference in GC time, but
you might have been close enough to heap saturation that vnodes tip you
over.

As an aside, one is likely to win very little net win from vnodes if one's
cluster is not now and will never be more than approximately 15 nodes.

=Rob

Re: Do vnodes need more memory?

[Tom van den Berge <to...@gmail.com>]

nodetool gcstat tells me this (the Total GC Elapsed is half or more of the
Interval).

We had to take the production load off the new vnode DC, since it was
messing things up badly. It means I'm not able to run any tools against it
at the moment.
The env.sh is default, and the servers have 8G ram.

It would be great if you could respond to my initial question though.
Thanks,
Tom

On Wed, Sep 23, 2015 at 4:14 PM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:

> This is interesting, where are you seeing that you're collecting 50% of
> the time? Is your env.sh the default? How much ram?
>
> Also, can you run this tool and send a minute worth of thread info:
>
> wget
> https://bintray.com/artifact/download/aragozin/generic/sjk-plus-0.3.6.jar
> java -jar sjk-plus-0.3.6.jar ttop -s localhost:7199 -n 30 -o CPU
> On Sep 23, 2015 7:09 AM, "Tom van den Berge" <to...@gmail.com>
> wrote:
>
>> I have two data centers, each with the same number of nodes, same
>> hardware (CPUs, memory), Cassandra version (2.1.6), replication factory,
>> etc. The only difference it that one data center uses vnodes, and the other
>> doesn't.
>>
>> The non-vnode DC works fine (and has been for a long time) under
>> production load: I'm seeing normal CPU and IO load and garbage collection
>> figures. But the vnode DC is struggling very hard under the same load. It
>> has been set up recently. The CPU load is very high, due to excessive
>> garbage collection (>50% of the time is spent collecting).
>>
>> So it seems that Cassandra simply doesn't have enough memory. I'm trying
>> to understand if this can be caused by the use of vnodes? Is there an
>> sensible reason why vnodes would consume more memory than regular nodes? Or
>> does any of you have the same experience? If not, I might be barking up the
>> wrong tree here, and I would love to know it before upgrading my servers
>> with more memory.
>>
>> Thanks,
>> Tom
>>
>

Re: Do vnodes need more memory?

[Sebastian Estevez <se...@datastax.com>]

This is interesting, where are you seeing that you're collecting 50% of the
time? Is your env.sh the default? How much ram?

Also, can you run this tool and send a minute worth of thread info:

wget
https://bintray.com/artifact/download/aragozin/generic/sjk-plus-0.3.6.jar
java -jar sjk-plus-0.3.6.jar ttop -s localhost:7199 -n 30 -o CPU
On Sep 23, 2015 7:09 AM, "Tom van den Berge" <to...@gmail.com>
wrote:

> I have two data centers, each with the same number of nodes, same hardware
> (CPUs, memory), Cassandra version (2.1.6), replication factory, etc. The
> only difference it that one data center uses vnodes, and the other doesn't.
>
> The non-vnode DC works fine (and has been for a long time) under
> production load: I'm seeing normal CPU and IO load and garbage collection
> figures. But the vnode DC is struggling very hard under the same load. It
> has been set up recently. The CPU load is very high, due to excessive
> garbage collection (>50% of the time is spent collecting).
>
> So it seems that Cassandra simply doesn't have enough memory. I'm trying
> to understand if this can be caused by the use of vnodes? Is there an
> sensible reason why vnodes would consume more memory than regular nodes? Or
> does any of you have the same experience? If not, I might be barking up the
> wrong tree here, and I would love to know it before upgrading my servers
> with more memory.
>
> Thanks,
> Tom
>