You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Curt Allred <cu...@mediosystems.com> on 2012/06/01 23:01:59 UTC
nodes moving spontaneously
We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times we have seen nodes move themselves to different locations in the ring. In each case, the move unbalanced the ring. In one case a node moved to the opposite side of the ring.
Sometime after the first spontaneous move we started using Datastax OpsCenter. The next 2 moves showed up in its event log like:
5/20/2012 11:23am - Info - Host 12.34.56.78 moved from '12345' to '54321'
where '12345' and '54321' are the old and new tokens.
Anyone know whats causing this?
Re: nodes moving spontaneously
Posted by Cyril Auburtin <cy...@gmail.com>.
There must be someone or something that triggered nodetool loadbalance or
nodetool move
Le 1 juin 2012 23:18, "Tyler Hobbs" <ty...@datastax.com> a écrit :
> OpsCenter just periodically calls describe_ring() on different nodes in
> the cluster, so that's how it's getting that information.
>
> Maybe try running nodetool ring on each node in your cluster to make sure
> they all have the same view of the ring?
>
> On Fri, Jun 1, 2012 at 4:01 PM, Curt Allred <cu...@mediosystems.com> wrote:
>
>> We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times
>> we have seen nodes move themselves to different locations in the ring. In
>> each case, the move unbalanced the ring. In one case a node moved to the
>> opposite side of the ring.****
>>
>> ** **
>>
>> Sometime after the first spontaneous move we started using Datastax
>> OpsCenter. The next 2 moves showed up in its event log like:****
>>
>> 5/20/2012 11:23am - Info - Host 12.34.56.78 moved from '12345' to '54321'
>> ****
>>
>> ** **
>>
>> where '12345' and '54321' are the old and new tokens.****
>>
>> ** **
>>
>> Anyone know whats causing this?****
>>
>> ** **
>>
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>
>
RE: nodes moving spontaneously
Posted by Curt Allred <cu...@mediosystems.com>.
Thanks for the tip. Checked "nodetool ring" on all nodes and they all have a consistent view of the ring. We have had other problems like nodes crashing etc so anything could have happened, but we're sure we didnt issue a "nodetool move" command.
From: Tyler Hobbs [mailto:tyler@datastax.com]
OpsCenter just periodically calls describe_ring() on different nodes in the cluster, so that's how it's getting that information.
Maybe try running nodetool ring on each node in your cluster to make sure they all have the same view of the ring?
Re: nodes moving spontaneously
Posted by Tyler Hobbs <ty...@datastax.com>.
OpsCenter just periodically calls describe_ring() on different nodes in the
cluster, so that's how it's getting that information.
Maybe try running nodetool ring on each node in your cluster to make sure
they all have the same view of the ring?
On Fri, Jun 1, 2012 at 4:01 PM, Curt Allred <cu...@mediosystems.com> wrote:
> We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times
> we have seen nodes move themselves to different locations in the ring. In
> each case, the move unbalanced the ring. In one case a node moved to the
> opposite side of the ring.****
>
> ** **
>
> Sometime after the first spontaneous move we started using Datastax
> OpsCenter. The next 2 moves showed up in its event log like:****
>
> 5/20/2012 11:23am - Info - Host 12.34.56.78 moved from '12345' to '54321'
> ****
>
> ** **
>
> where '12345' and '54321' are the old and new tokens.****
>
> ** **
>
> Anyone know whats causing this?****
>
> ** **
>
--
Tyler Hobbs
DataStax <http://datastax.com/>
Re: nodes moving spontaneously
Posted by Brandon Williams <dr...@gmail.com>.
Nodes don't move themselves, you likely have some kind of 'bouncing
gossip' issue where a node was removed/replaced and is hanging around,
but only periodically held in statee between nodes. Unfortunately
node removal is very prone to this before 0.8.3 and even after that,
you can't fix it without
https://issues.apache.org/jira/browse/CASSANDRA-3337 so upgrading is
really your best bet here.
On Fri, Jun 1, 2012 at 4:01 PM, Curt Allred <cu...@mediosystems.com> wrote:
> We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times we
> have seen nodes move themselves to different locations in the ring. In each
> case, the move unbalanced the ring. In one case a node moved to the opposite
> side of the ring.
>
>
>
> Sometime after the first spontaneous move we started using Datastax
> OpsCenter. The next 2 moves showed up in its event log like:
>
> 5/20/2012 11:23am - Info - Host 12.34.56.78 moved from '12345' to '54321'
>
>
>
> where '12345' and '54321' are the old and new tokens.
>
>
>
> Anyone know whats causing this?
>
>