You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nimi Wariboko Jr <ni...@gmail.com> on 2013/09/19 01:26:12 UTC
Rebalancing vnodes cluster
Hi,
When I started with cassandra I had originally set it up to use tokens. I
then migrated to vnodes (using shuffle), but my cluster isn't balanced (http://imgur.com/73eNhJ3).
What steps can I take to balance my cluster?
Thanks,
Nimi
Re: Rebalancing vnodes cluster
Posted by Nimi Wariboko Jr <ni...@gmail.com>.
This isn't the case. I noticed the error because of some unusual hotspotting. `nodetool status` also shows the cluster is unbalanced.
root@cass1:~# nodetool status
Datacenter: 129
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.129.196.6 140.06 GB 269 21.8% ce8d4ea9-8106-44b3-a2dd-c0230eb53c94 196
UN 10.129.196.21 157.84 GB 256 22.4% d7009278-3191-48a1-8112-dc350e78b783 196
UN 10.129.196.4 138.52 GB 260 23.4% cde6c3be-a066-47f2-abc2-b1d78bee0d7c 196
UN 10.129.196.5 261.46 GB 239 32.3% 2cb24510-2f89-46b2-96b9-873f8e8e50da 196
On Wednesday, September 18, 2013 at 4:53 PM, Nick Bailey wrote:
> OpsCenter only supports vnodes minimally at this point.
>
> More specifically, it chooses a random token that a node owns in order to display that node on the ring. So a vnode cluster will always appear unbalanced in OpsCenter. Your cluster is probably balanced fine, but 'nodetoo status' should indicate that.
>
> Better vnode support is coming up in a future OpsCenter release.
>
>
> On Wed, Sep 18, 2013 at 6:26 PM, Nimi Wariboko Jr <nimiwaribokoj@gmail.com (mailto:nimiwaribokoj@gmail.com)> wrote:
> > Hi,
> >
> > When I started with cassandra I had originally set it up to use tokens. I
> > then migrated to vnodes (using shuffle), but my cluster isn't balanced (http://imgur.com/73eNhJ3).
> >
> > What steps can I take to balance my cluster?
> >
> > Thanks,
> > Nimi
> >
>
>
>
Re: Rebalancing vnodes cluster
Posted by Nick Bailey <ni...@datastax.com>.
OpsCenter only supports vnodes minimally at this point.
More specifically, it chooses a random token that a node owns in order to
display that node on the ring. So a vnode cluster will always appear
unbalanced in OpsCenter. Your cluster is probably balanced fine, but
'nodetoo status' should indicate that.
Better vnode support is coming up in a future OpsCenter release.
On Wed, Sep 18, 2013 at 6:26 PM, Nimi Wariboko Jr
<ni...@gmail.com>wrote:
> Hi,
>
> When I started with cassandra I had originally set it up to use tokens. I
> then migrated to vnodes (using shuffle), but my cluster isn't balanced (
> http://imgur.com/73eNhJ3).
>
> What steps can I take to balance my cluster?
>
> Thanks,
> Nimi
>
Re: Rebalancing vnodes cluster
Posted by Nimi Wariboko Jr <ni...@gmail.com>.
We had originally started with 3 nodes w/ 32GB ram and 768GB SSDs. I pretty much Google'd my way into setting up cassandra and set it up using tokens because I was following an older docco. We were using Cassandra 1.2.5, I learned about vnodes later on and regretted waking up that morning.
1.) I'm not sure if shuffle was successful. We started shuffling on Jun 7th and killed it on the 17th. We let it run over 2 weekends (10 days) and it the node shuffle tool didn't report any meaningful progress. I explained this over IRC and was told `node shuffle` takes a really long time and you shouldn't use it. At the time our ring looked "mostly" balanced so we just killed it. We were migrating from a MongoDB cluster and didn't want to pay for 2 clusters.
2.) During the shuffle we had upped our RF to 2, did not a do a repair and lost 1/3rd of our data. Fortunately we could just use sstable tool to reload the data as it was really deleted.
3.) We ran cleanup a couple days later
4.) Cassandra 1.2.5
After all this, we converted another mongo node we had into Cassandra (same specs) for a cluster of size 4. Now after 4 months, one node (the subject of this thread) is growing faster than the others (which is leading to hot spotting as well). I guess this has to do with the unfinished shuffle? Are there any remedies for this?
On Thursday, September 19, 2013 at 9:50 AM, Robert Coli wrote:
> On Wed, Sep 18, 2013 at 4:26 PM, Nimi Wariboko Jr <nimiwaribokoj@gmail.com (mailto:nimiwaribokoj@gmail.com)> wrote:
> > When I started with cassandra I had originally set it up to use tokens. I
> > then migrated to vnodes (using shuffle), but my cluster isn't balanced (http://imgur.com/73eNhJ3).
>
> Are you saying that (other than the imbalance that is the subject of this thread) you were able to use "shuffle" successfully on a cluster with ~150gb per node?
>
> 1) How long did it take?
> 2) Did you experience any difficulties while doing so?
> 3) Have you run cleanup yet?
> 4) What version of Cassandra?
>
> =Rob
>
>
>
>
>
>
Re: Rebalancing vnodes cluster
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Sep 18, 2013 at 4:26 PM, Nimi Wariboko Jr
<ni...@gmail.com>wrote:
> When I started with cassandra I had originally set it up to use tokens. I
> then migrated to vnodes (using shuffle), but my cluster isn't balanced (
> http://imgur.com/73eNhJ3).
>
Are you saying that (other than the imbalance that is the subject of this
thread) you were able to use "shuffle" successfully on a cluster with
~150gb per node?
1) How long did it take?
2) Did you experience any difficulties while doing so?
3) Have you run cleanup yet?
4) What version of Cassandra?
=Rob