You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jouni Hartikainen <jo...@reaktor.fi> on 2013/02/09 13:15:47 UTC
Bootstrapping a new node to a virtual node cluster
Hello all,
I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by adding a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration for the new node as well.
My cluster looks like this before adding the new node:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.154.11 1.49 GB 256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1
UN 192.168.154.12 1.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1
UN 192.168.154.13 1.64 GB 256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1
And corresponding gossipinfo:
/192.168.154.12
RPC_ADDRESS:192.168.154.12
DC:datacenter1
STATUS:NORMAL,-1072164398478041156
LOAD:1.719425018E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
/192.168.154.11
RPC_ADDRESS:192.168.154.11
DC:datacenter1
STATUS:NORMAL,-1158837144480089281
LOAD:1.514343678E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:234b82a4-3812-4261-adab-deb805942d63
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
/192.168.154.13
RPC_ADDRESS:192.168.154.13
DC:datacenter1
STATUS:NORMAL,-1135137292201587328
LOAD:1.765093695E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
I have now set the correct net addresses & seeds in the cassandra.yaml of the new node (.14) and then started it with num_tokens set to 256 and initial_token commented out. Everything seems to go OK as I get the following prints on the log:
On node 192.168.154.11:
INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node /192.168.154.14 is now part of the cluster
INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) InetAddress /192.168.154.14 is now UP
INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning transfer to /192.168.154.14
And on node 192.168.154.14 (the new node):
INFO 12:30:26,843 Loading persisted ring state
INFO 12:30:26,846 Starting up server gossip
WARN 12:30:26,853 No host ID found, created a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This should happen exactly once per node).
INFO 12:30:26,979 Starting Messaging Service on port 7000
INFO 12:30:27,014 JOINING: waiting for ring information
INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster
INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP
INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster
INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP
INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster
INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP
INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap
INFO 12:30:29,015 JOINING: getting bootstrap token
INFO 12:30:29,157 JOINING: sleeping 30000 ms for pending range setup
INFO 12:30:59,159 JOINING: Starting to bootstrap...
However, the new node does not show up in "nodetool status" (even if queried from the new node itself):
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.154.11 1.49 GB 256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1
UN 192.168.154.12 1.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1
UN 192.168.154.13 1.64 GB 256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1
It shows up in the gossip still:
/192.168.154.12
RPC_ADDRESS:192.168.154.12
DC:datacenter1
STATUS:NORMAL,-1072164398478041156
LOAD:1.719430632E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
SEVERITY:0.0
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.14
RPC_ADDRESS:192.168.154.14
DC:datacenter1
STATUS:BOOT,8077752099299332137
LOAD:105101.0
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.11
RPC_ADDRESS:192.168.154.11
DC:datacenter1
STATUS:NORMAL,-1158837144480089281
LOAD:1.596505929E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:234b82a4-3812-4261-adab-deb805942d63
SEVERITY:0.0
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.13
RPC_ADDRESS:192.168.154.13
DC:datacenter1
STATUS:NORMAL,-1135137292201587328
LOAD:1.765062944E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
SEVERITY:-6.776263578034403E-21
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
I also noticed that the new node is not listening for its thrift port (9160) until all data streaming has ended.
Couple of questions after all this:
1) Is the new node not showing up in the "nodetool status" expected behavior?
2) Do I have to wait for all the data streaming to end until the new node starts to accept writes? If yes, has this changed from 1.1? (I remember nodes becoming writeable immediately after booting, but readable only after streaming has ended)
3) "nodetool info" on the new node raises an exception "Exception in thread "main" java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)" This is probably not normal?
4) Is it so that the virtual tokens for the new node are not assigned before the data streaming ends? ("select * from peers;" on .11 returns null for the tokens of .14) If so, how the existing nodes decide what data to stream to a new node?
-Jouni
Re: Bootstrapping a new node to a virtual node cluster
Posted by Jouni Hartikainen <jo...@reaktor.fi>.
On Feb 12, 2013, at 18:58 , aaron morton <aa...@thelastpickle.com> wrote:
> Just checking if this sorted it's self out?
Well, partially. :) Situation at the moment:
>> 1) Is the new node not showing up in the "nodetool status" expected behavior?
Not showing still, don't know if should. :)
>> 2) Do I have to wait for all the data streaming to end until the new node starts to accept writes? If yes, has this changed from 1.1? (I remember nodes becoming writeable immediately after booting, but readable only after streaming has ended)
This does not seem to be the case. Writes are indeed relayed to the bootstrapping node during the bootstrapping even if it is not shown as a part of the cluster.
>> 3) "nodetool info" on the new node raises an exception "Exception in thread "main" java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)" This is probably not normal?
Still an issue. Should probably file a bug on this one?
>> 4) Is it so that the virtual tokens for the new node are not assigned before the data streaming ends? ("select * from peers;" on .11 returns null for the tokens of .14) If so, how the existing nodes decide what data to stream to a new node?
By reading the code it seems that the tokens are indeed generated early in the bootstrapping sequence and propagated (via gossip?) to the other nodes. However, I haven't still figured out why they are not shown in the list of peers when querying using CQL.
FYI.. While playing with this I bumbed into a bug that prevents compactions from running while a node is in the bootstrapping state: https://issues.apache.org/jira/browse/CASSANDRA-5244 Might be an issue at least for those running clusters with lots of data per node (= long bootstrap times) and heavy write load (= lots of data to compacted).
Cheers,
-Jouni
Re: Bootstrapping a new node to a virtual node cluster
Posted by aaron morton <aa...@thelastpickle.com>.
Just checking if this sorted it's self out?
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 10/02/2013, at 1:15 AM, Jouni Hartikainen <jo...@reaktor.fi> wrote:
> Hello all,
>
> I have a cluster of three nodes running 1.2.1 and I'd like to increase the capacity by adding a new node. I'm using virtual nodes with 256 tokens and planning to use the same configuration for the new node as well.
>
> My cluster looks like this before adding the new node:
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 192.168.154.11 1.49 GB 256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1
> UN 192.168.154.12 1.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1
> UN 192.168.154.13 1.64 GB 256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1
>
>
> And corresponding gossipinfo:
>
> /192.168.154.12
> RPC_ADDRESS:192.168.154.12
> DC:datacenter1
> STATUS:NORMAL,-1072164398478041156
> LOAD:1.719425018E9
> SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
> NET_VERSION:6
> HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
> SEVERITY:0.0
> RELEASE_VERSION:1.2.1
> RACK:rack1
> /192.168.154.11
> RPC_ADDRESS:192.168.154.11
> DC:datacenter1
> STATUS:NORMAL,-1158837144480089281
> LOAD:1.514343678E9
> SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
> NET_VERSION:6
> HOST_ID:234b82a4-3812-4261-adab-deb805942d63
> SEVERITY:0.0
> RELEASE_VERSION:1.2.1
> RACK:rack1
> /192.168.154.13
> RPC_ADDRESS:192.168.154.13
> DC:datacenter1
> STATUS:NORMAL,-1135137292201587328
> LOAD:1.765093695E9
> SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
> NET_VERSION:6
> HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
> SEVERITY:0.0
> RELEASE_VERSION:1.2.1
> RACK:rack1
>
>
> I have now set the correct net addresses & seeds in the cassandra.yaml of the new node (.14) and then started it with num_tokens set to 256 and initial_token commented out. Everything seems to go OK as I get the following prints on the log:
>
> On node 192.168.154.11:
>
> INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node /192.168.154.14 is now part of the cluster
> INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) InetAddress /192.168.154.14 is now UP
> INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning transfer to /192.168.154.14
>
> And on node 192.168.154.14 (the new node):
>
> INFO 12:30:26,843 Loading persisted ring state
> INFO 12:30:26,846 Starting up server gossip
> WARN 12:30:26,853 No host ID found, created a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This should happen exactly once per node).
> INFO 12:30:26,979 Starting Messaging Service on port 7000
> INFO 12:30:27,014 JOINING: waiting for ring information
> INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster
> INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP
> INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster
> INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP
> INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster
> INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP
> INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap
> INFO 12:30:29,015 JOINING: getting bootstrap token
> INFO 12:30:29,157 JOINING: sleeping 30000 ms for pending range setup
> INFO 12:30:59,159 JOINING: Starting to bootstrap...
>
> However, the new node does not show up in "nodetool status" (even if queried from the new node itself):
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 192.168.154.11 1.49 GB 256 100.0% 234b82a4-3812-4261-adab-deb805942d63 rack1
> UN 192.168.154.12 1.6 GB 256 100.0% 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1
> UN 192.168.154.13 1.64 GB 256 100.0% 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1
>
> It shows up in the gossip still:
>
> /192.168.154.12
> RPC_ADDRESS:192.168.154.12
> DC:datacenter1
> STATUS:NORMAL,-1072164398478041156
> LOAD:1.719430632E9
> SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
> NET_VERSION:6
> HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
> SEVERITY:0.0
> RELEASE_VERSION:1.2.1-SNAPSHOT
> RACK:rack1
> /192.168.154.14
> RPC_ADDRESS:192.168.154.14
> DC:datacenter1
> STATUS:BOOT,8077752099299332137
> LOAD:105101.0
> SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
> NET_VERSION:6
> HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110
> RELEASE_VERSION:1.2.1-SNAPSHOT
> RACK:rack1
> /192.168.154.11
> RPC_ADDRESS:192.168.154.11
> DC:datacenter1
> STATUS:NORMAL,-1158837144480089281
> LOAD:1.596505929E9
> SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
> NET_VERSION:6
> HOST_ID:234b82a4-3812-4261-adab-deb805942d63
> SEVERITY:0.0
> RELEASE_VERSION:1.2.1-SNAPSHOT
> RACK:rack1
> /192.168.154.13
> RPC_ADDRESS:192.168.154.13
> DC:datacenter1
> STATUS:NORMAL,-1135137292201587328
> LOAD:1.765062944E9
> SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
> NET_VERSION:6
> HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
> SEVERITY:-6.776263578034403E-21
> RELEASE_VERSION:1.2.1-SNAPSHOT
> RACK:rack1
>
>
> I also noticed that the new node is not listening for its thrift port (9160) until all data streaming has ended.
>
>
> Couple of questions after all this:
> 1) Is the new node not showing up in the "nodetool status" expected behavior?
> 2) Do I have to wait for all the data streaming to end until the new node starts to accept writes? If yes, has this changed from 1.1? (I remember nodes becoming writeable immediately after booting, but readable only after streaming has ended)
> 3) "nodetool info" on the new node raises an exception "Exception in thread "main" java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)" This is probably not normal?
> 4) Is it so that the virtual tokens for the new node are not assigned before the data streaming ends? ("select * from peers;" on .11 returns null for the tokens of .14) If so, how the existing nodes decide what data to stream to a new node?
>
>
> -Jouni