You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by ai...@iponweb.net on 2014/10/25 14:36:29 UTC
Bootstrapping new node overloads cpu on existing nodes
Dear all,
So, here is our setup so far:
- Ubuntu 12.04
- Cassandra 2.0.10, JDK 1.7.0_65-b17
- 6 nodes (EC2 c3.8xlarge/ 32 cores/60GB RAM, EBS disks for data,
ephemeral SSD for commit logs etc)
- pretty heavy write load - 100Ks/second
- RF=2, one dc, 2 racks
- everything works just fine with low CPU consumption - load average tends to be around 4-10
Now, we are trying to add a node. This cases a heavy load on existing nodes - like over 100 load average. The cluster becomes unresponsive, writes and reads mostly fail.
The weird observations are that:
- without adding new node CPU is low
- if we turn off writes while adding a new node load average on existing nodes drops back to 4-10 and the new node just fine
I've checked VisualVM sampling and basically all the CPU on existing nodes is consumed by org.jboss.netty.channel.socket.nio.SelectorUtil.select().
What we tried so far:
- throttling streaming - no impact
- disabling internode compression - no impact
- disabling autocompaction on existing nodes - no impact
- even running with -Dorg.jboss.netty.epollBugWorkaround=true - no impact
And as of now we are somewhat desperate as this behavior is a blocker for us - we can't afford losing writes and we will need to expand C* dynamically.
Anyone has encountered something similar? Any ideas/hints? Thanks
Re: Bootstrapping new node overloads cpu on existing nodes
Posted by ai...@iponweb.net.
No, simply bootstrapping. Basically, as soon as the new node starts joining load average spikes like x10
And it's not like the load comes from streaming (judging from what I see in visualvm), it's writes that start taking much more cpu, mostly for nio socket select.
От: Jens Rantil
Отправлено: суббота, 25 октября 2014 г., 21:22
Кому: user@cassandra.apache.org
Ответить: user@cassandra.apache.org
Тема: Re: Bootstrapping new node overloads cpu on existing nodes
Just to clarify, does "adding node" include initiating a repair for the cluster? Or you are simply bootstrapping a new node, nothing else?
—
Sent from Mailbox
On Sat, Oct 25, 2014 at 2:38 PM, aivanov@iponweb.net <ai...@iponweb.net> wrote:
Dear all,
So, here is our setup so far:
- Ubuntu 12.04
- Cassandra 2.0.10, JDK 1.7.0_65-b17
- 6 nodes (EC2 c3.8xlarge/ 32 cores/60GB RAM, EBS disks for data,
ephemeral SSD for commit logs etc)
- pretty heavy write load - 100Ks/second
- RF=2, one dc, 2 racks
- everything works just fine with low CPU consumption - load average tends to be around 4-10
Now, we are trying to add a node. This cases a heavy load on existing nodes - like over 100 load average. The cluster becomes unresponsive, writes and reads mostly fail.
The weird observations are that:
- without adding new node CPU is low
- if we turn off writes while adding a new node load average on existing nodes drops back to 4-10 and the new node just fine
I've checked VisualVM sampling and basically all the CPU on existing nodes is consumed by org.jboss.netty.channel.socket.nio.SelectorUtil.select().
What we tried so far:
- throttling streaming - no impact
- disabling internode compression - no impact
- disabling autocompaction on existing nodes - no impact
- even running with -Dorg.jboss.netty.epollBugWorkaround=true - no impact
And as of now we are somewhat desperate as this behavior is a blocker for us - we can't afford losing writes and we will need to expand C* dynamically.
Anyone has encountered something similar? Any ideas/hints? Thanks
Re: Bootstrapping new node overloads cpu on existing nodes
Posted by Jens Rantil <je...@tink.se>.
Just to clarify, does "adding node" include initiating a repair for the cluster? Or you are simply bootstrapping a new node, nothing else?
—
Sent from Mailbox
On Sat, Oct 25, 2014 at 2:38 PM, null <ai...@iponweb.net> wrote:
> Dear all,
> So, here is our setup so far:
> - Ubuntu 12.04
> - Cassandra 2.0.10, JDK 1.7.0_65-b17
> - 6 nodes (EC2 c3.8xlarge/ 32 cores/60GB RAM, EBS disks for data,
> ephemeral SSD for commit logs etc)
> - pretty heavy write load - 100Ks/second
> - RF=2, one dc, 2 racks
> - everything works just fine with low CPU consumption - load average tends to be around 4-10
> Now, we are trying to add a node. This cases a heavy load on existing nodes - like over 100 load average. The cluster becomes unresponsive, writes and reads mostly fail.
> The weird observations are that:
> - without adding new node CPU is low
> - if we turn off writes while adding a new node load average on existing nodes drops back to 4-10 and the new node just fine
> I've checked VisualVM sampling and basically all the CPU on existing nodes is consumed by org.jboss.netty.channel.socket.nio.SelectorUtil.select().
> What we tried so far:
> - throttling streaming - no impact
> - disabling internode compression - no impact
> - disabling autocompaction on existing nodes - no impact
> - even running with -Dorg.jboss.netty.epollBugWorkaround=true - no impact
> And as of now we are somewhat desperate as this behavior is a blocker for us - we can't afford losing writes and we will need to expand C* dynamically.
> Anyone has encountered something similar? Any ideas/hints? Thanks