You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Christopher Wirt <ch...@struq.com> on 2013/08/05 13:30:44 UTC
Reducing the number of vnodes
Hi,
I'm thinking about reducing the number of vnodes per server.
We have 3 DC setup - one with 9 nodes, two with 3 nodes each.
Each node has 256 vnodes. We've found that repair operations are beginning
to take too long.
Is reducing the number of vnodes to 64/32 likely to help our situation?
What options do I have for achieving this in a live cluster?
Thanks,
Chris
Re: Reducing the number of vnodes
Posted by Richard Low <ri...@wentnet.com>.
On 5 August 2013 12:30, Christopher Wirt <ch...@struq.com> wrote:
I’m thinking about reducing the number of vnodes per server.****
>
> ** **
>
> We have 3 DC setup – one with 9 nodes, two with 3 nodes each.****
>
> ** **
>
> Each node has 256 vnodes. We’ve found that repair operations are beginning
> to take too long.****
>
> ** **
>
> Is reducing the number of vnodes to 64/32 likely to help our situation?
>
Unlikely. The amount of time repair takes only depends on the number of
vnodes if you have a tiny amount of data. If you have 256 vnodes and not
much more than 256 * num_nodes rows, the overhead of splitting up the
repairs into separate ranges for each vnode is significant. However, once
your dataset becomes bigger than this trivial amount, the vnode overhead of
repair becomes totally insignificant.
The main reasons for slow repair are lots of data or lots of data out of
sync. You can tell how much is out of sync by looking in the logs - it
will say how many ranges within each vnode range need repairing.
Richard.
Re: Reducing the number of vnodes
Posted by Aaron Morton <aa...@thelastpickle.com>.
Repair runs in two phases, first it works out the differences then it streams the data. The length of the first depends on the size of the data and the second on the level of inconsistency.
To track the first use nodetool compaction stats or look in the logs for the messages about requesting or receiving Merkle Tree's.
To track the second use nodetool netstats or look in the logs for messages about streaming X number of ranges.
Hope that helps.
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 6/08/2013, at 1:41 AM, Christopher Wirt <ch...@struq.com> wrote:
> 1.2.4. Really hesitant to upgrade versions due to the inevitable issues it will cause.
>
> Guess I could upgrade a single node and let that run for a while before upgrading all nodes.
>
> From: Haithem Jarraya [mailto:a-hjarraya@expedia.com]
> Sent: 05 August 2013 13:04
> To: user@cassandra.apache.org
> Subject: Re: Reducing the number of vnodes
>
> Chris,
> Which C* version are you running?
> You might want to do an upgrade to the latest version before reducing the vnode counts, a lot of fixes and improvement went in lately, it might help you getting your repair faster.
>
> H
>
> On 5 Aug 2013, at 12:30, Christopher Wirt <ch...@struq.com> wrote:
>
>
> Hi,
>
> I’m thinking about reducing the number of vnodes per server.
>
> We have 3 DC setup – one with 9 nodes, two with 3 nodes each.
>
> Each node has 256 vnodes. We’ve found that repair operations are beginning to take too long.
>
> Is reducing the number of vnodes to 64/32 likely to help our situation?
> What options do I have for achieving this in a live cluster?
>
>
> Thanks,
>
> Chris
RE: Reducing the number of vnodes
Posted by Christopher Wirt <ch...@struq.com>.
1.2.4. Really hesitant to upgrade versions due to the inevitable issues it
will cause.
Guess I could upgrade a single node and let that run for a while before
upgrading all nodes.
From: Haithem Jarraya [mailto:a-hjarraya@expedia.com]
Sent: 05 August 2013 13:04
To: user@cassandra.apache.org
Subject: Re: Reducing the number of vnodes
Chris,
Which C* version are you running?
You might want to do an upgrade to the latest version before reducing the
vnode counts, a lot of fixes and improvement went in lately, it might help
you getting your repair faster.
H
On 5 Aug 2013, at 12:30, Christopher Wirt <ch...@struq.com> wrote:
Hi,
I'm thinking about reducing the number of vnodes per server.
We have 3 DC setup - one with 9 nodes, two with 3 nodes each.
Each node has 256 vnodes. We've found that repair operations are beginning
to take too long.
Is reducing the number of vnodes to 64/32 likely to help our situation?
What options do I have for achieving this in a live cluster?
Thanks,
Chris
Re: Reducing the number of vnodes
Posted by Haithem Jarraya <a-...@expedia.com>.
Chris,
Which C* version are you running?
You might want to do an upgrade to the latest version before reducing the vnode counts, a lot of fixes and improvement went in lately, it might help you getting your repair faster.
H
On 5 Aug 2013, at 12:30, Christopher Wirt <ch...@struq.com>> wrote:
Hi,
I’m thinking about reducing the number of vnodes per server.
We have 3 DC setup – one with 9 nodes, two with 3 nodes each.
Each node has 256 vnodes. We’ve found that repair operations are beginning to take too long.
Is reducing the number of vnodes to 64/32 likely to help our situation?
What options do I have for achieving this in a live cluster?
Thanks,
Chris
Re: Reducing the number of vnodes
Posted by Radim Kolar <hs...@filez.com>.
> Is reducing the number of vnodes to 64/32 likely to help our situation?
>
with just 3 nodes per datacenter reduce vnodes to 1.
>
> What options do I have for achieving this in a live cluster?
>
you need to remove node, move its data to other 2 and add it with
different vnodes count.