You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Andrew Cobley <a....@dundee.ac.uk> on 2013/08/14 21:02:17 UTC
Vnodes, adding a node ?
I have small test cluster of 2 nodes. I ran a stress test on it and with nodetool status received the following:
/usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.0.11 141.13 MB 256 49.2% 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
UN 192.168.0.10 145.59 MB 256 50.8% 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
I then added a third node with no machines writing to the system. Using nodetool status I got the following:
/usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.0.11 141.12 MB 256 32.2% 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
UN 192.168.0.10 145.59 MB 256 35.3% 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
UN 192.168.0.12 111.9 KB 256 32.5% e5e6d8bd-c652-4c18-8fa3-3d71471eee65 rack1
Is this correct ? I was under the impression that adding a node to an existing cluster would distribute the load around the cluster. Am I perhaps missing a step or have a config error perhaps ?
Ta
Andy
The University of Dundee is a registered Scottish Charity, No: SC015096
Re: Vnodes, adding a node ?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Aug 15, 2013 at 11:02 AM, rash aroskar <ra...@gmail.com>wrote:
> what do you mean by not adding as a seed? if I add new node to existing
> cluster, the new node should not be added as a seed in cassandra.yaml for
> other nodes in the ring?
> when should it be added as a seed then? once the cluster is balanced? or
> after manually running rebuild command?
>
My impression is that the project doesn't know the answer to your
questions. If they do, it seems that they do not care to publicly clarify
the logic and best practice for state management with regard to seed nodes
and bootstrap.
https://issues.apache.org/jira/browse/CASSANDRA-5836
The only thing I know for certain is that the current situation seems
illogical : if a given node is defined as a seed in its own conf file, it
will refuse to bootstrap. But it might be defined as a seed in other nodes
conf files, so removing it from its own seed list seems unlikely to prevent
it from "being a seed." It seems illogical that a "seed" should be
prevented from bootstrapping if it can bootstrap just fine... as long as it
doesn't know it's a seed.
=Rob
Re: Vnodes, adding a node ?
Posted by rash aroskar <ra...@gmail.com>.
what do you mean by not adding as a seed? if I add new node to existing
cluster, the new node should not be added as a seed in cassandra.yaml for
other nodes in the ring?
when should it be added as a seed then? once the cluster is balanced? or
after manually running rebuild command?
On Wed, Aug 14, 2013 at 3:34 PM, Andrew Cobley <a....@dundee.ac.uk>wrote:
> That looks like the problem. I added the node with that machine as a
> seed, realized my mistake and restarted the machine with the correct seed.
> it joined the ring but without streaming. Nodetool rebuild however doesn't
> seem to be fixing the situation.
>
> I'll remove the node and try re-adding it after cleaning
> /var/lib/cassandra.
>
> Andy
>
>
>
> ------------------------------
> *From:* Richard Low [richard@wentnet.com]
> *Sent:* 14 August 2013 20:11
> *To:* user@cassandra.apache.org
> *Subject:* Re: Vnodes, adding a node ?
>
> On 14 August 2013 20:02, Andrew Cobley <a....@dundee.ac.uk> wrote:
>
>> I have small test cluster of 2 nodes. I ran a stress test on it and
>> with nodetool status received the following:
>>
>> /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UN 192.168.0.11 141.13 MB 256 49.2%
>> 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
>> UN 192.168.0.10 145.59 MB 256 50.8%
>> 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
>>
>> I then added a third node with no machines writing to the system. Using
>> nodetool status I got the following:
>>
>> /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UN 192.168.0.11 141.12 MB 256 32.2%
>> 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
>> UN 192.168.0.10 145.59 MB 256 35.3%
>> 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
>> UN 192.168.0.12 111.9 KB 256 32.5%
>> e5e6d8bd-c652-4c18-8fa3-3d71471eee65 rack1
>>
>> Is this correct ? I was under the impression that adding a node to an
>> existing cluster would distribute the load around the cluster. Am I perhaps
>> missing a step or have a config error perhaps ?
>>
>
> How did you add the node? It looks like it didn't bootstrap but just
> joined the ring. You need to make sure the node is not set as a seed and
> that auto_bootstrap is true (the default).
>
> Alternatively, you could run 'nodetool rebuild' to stream data from the
> other nodes.
>
> Richard.
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
RE: Vnodes, adding a node ?
Posted by Andrew Cobley <a....@dundee.ac.uk>.
That looks like the problem. I added the node with that machine as a seed, realized my mistake and restarted the machine with the correct seed. it joined the ring but without streaming. Nodetool rebuild however doesn't seem to be fixing the situation.
I'll remove the node and try re-adding it after cleaning /var/lib/cassandra.
Andy
________________________________
From: Richard Low [richard@wentnet.com]
Sent: 14 August 2013 20:11
To: user@cassandra.apache.org
Subject: Re: Vnodes, adding a node ?
On 14 August 2013 20:02, Andrew Cobley <a....@dundee.ac.uk>> wrote:
I have small test cluster of 2 nodes. I ran a stress test on it and with nodetool status received the following:
/usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.0.11 141.13 MB 256 49.2% 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
UN 192.168.0.10 145.59 MB 256 50.8% 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
I then added a third node with no machines writing to the system. Using nodetool status I got the following:
/usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.0.11 141.12 MB 256 32.2% 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
UN 192.168.0.10 145.59 MB 256 35.3% 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
UN 192.168.0.12 111.9 KB 256 32.5% e5e6d8bd-c652-4c18-8fa3-3d71471eee65 rack1
Is this correct ? I was under the impression that adding a node to an existing cluster would distribute the load around the cluster. Am I perhaps missing a step or have a config error perhaps ?
How did you add the node? It looks like it didn't bootstrap but just joined the ring. You need to make sure the node is not set as a seed and that auto_bootstrap is true (the default).
Alternatively, you could run 'nodetool rebuild' to stream data from the other nodes.
Richard.
The University of Dundee is a registered Scottish Charity, No: SC015096
Re: Vnodes, adding a node ?
Posted by Richard Low <ri...@wentnet.com>.
On 14 August 2013 20:02, Andrew Cobley <a....@dundee.ac.uk> wrote:
> I have small test cluster of 2 nodes. I ran a stress test on it and with
> nodetool status received the following:
>
> /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 192.168.0.11 141.13 MB 256 49.2%
> 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
> UN 192.168.0.10 145.59 MB 256 50.8%
> 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
>
> I then added a third node with no machines writing to the system. Using
> nodetool status I got the following:
>
> /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 192.168.0.11 141.12 MB 256 32.2%
> 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
> UN 192.168.0.10 145.59 MB 256 35.3%
> 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
> UN 192.168.0.12 111.9 KB 256 32.5%
> e5e6d8bd-c652-4c18-8fa3-3d71471eee65 rack1
>
> Is this correct ? I was under the impression that adding a node to an
> existing cluster would distribute the load around the cluster. Am I perhaps
> missing a step or have a config error perhaps ?
>
How did you add the node? It looks like it didn't bootstrap but just
joined the ring. You need to make sure the node is not set as a seed and
that auto_bootstrap is true (the default).
Alternatively, you could run 'nodetool rebuild' to stream data from the
other nodes.
Richard.
Re: Vnodes, adding a node ?
Posted by sankalp kohli <ko...@gmail.com>.
What steps did you follow to add the 3 node?
On Wed, Aug 14, 2013 at 12:02 PM, Andrew Cobley <a....@dundee.ac.uk>wrote:
> I have small test cluster of 2 nodes. I ran a stress test on it and with
> nodetool status received the following:
>
> /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 192.168.0.11 141.13 MB 256 49.2%
> 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
> UN 192.168.0.10 145.59 MB 256 50.8%
> 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
>
> I then added a third node with no machines writing to the system. Using
> nodetool status I got the following:
>
> /usr/local/bin/apache-cassandra-2.0.0-rc1/log $ ../bin/nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 192.168.0.11 141.12 MB 256 32.2%
> 4d281e2e-efd9-4abf-bb70-ebdf8e2b4fc3 rack1
> UN 192.168.0.10 145.59 MB 256 35.3%
> 7fc5795a-bd1b-4e42-88d6-024c5216a893 rack1
> UN 192.168.0.12 111.9 KB 256 32.5%
> e5e6d8bd-c652-4c18-8fa3-3d71471eee65 rack1
>
> Is this correct ? I was under the impression that adding a node to an
> existing cluster would distribute the load around the cluster. Am I perhaps
> missing a step or have a config error perhaps ?
>
>
> Ta
> Andy
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
>