You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jean-Armel Luce <ja...@gmail.com> on 2013/02/21 15:44:55 UTC
Adding new nodes in a cluster with virtual nodes
Hello,
We are using Cassandra 1.2.0.
We have a cluster of 16 physical nodes, each node has 256 virtual nodes.
We want to add 2 new nodes in our cluster : we follow the procedure as
explained here :
http://www.datastax.com/docs/1.2/operations/add_replace_nodes.
After starting 1 of the new node, we can see that this new node has 256
tokens ==>looks good
We can see that this node is in the ring (using nodetool status) ==> looks
good
After the bootstrap is finished in the new node, no data has been moved
automatically from the old nodes to this new node.
However, when we send insert queries in our cluster, the new node accepts
to insert the new rows.
Please, could you tell me if we need to perform a nodetool repair after the
bootstrap of the new node ?
What happens if we perform a nodetool cleanup in the old nodes before doing
the nodetool repair ? (Is there a risk of loosing some data ?)
Regards.
Jean Armel
Re: Adding new nodes in a cluster with virtual nodes
Posted by Jean-Armel Luce <ja...@gmail.com>.
Hi Aaron,
I tried again to add a node in the cluster. This time, I added the new node
in the seeds list after the bootstrap (the first time, I added the new node
in the seeds list before the bootstrap).
And it works !!!
Thanks Aaron.
Regards.
Jean Armel.
2013/2/22 Jean-Armel Luce <ja...@gmail.com>
> Thanks Aaron.
>
> I shall investigate more next week about this.
>
> regards.
>
> Jean Armel
>
>
> 2013/2/22 aaron morton <aa...@thelastpickle.com>
>
>> So, it looks that the repair is required if we want to add new nodes in
>> our platform, but I don't understand why.
>>
>> Bootstrapping should take care of it. But new seed nodes do not
>> bootstrap. Check the logs on the nodes you added to see what messages have
>> "bootstrap" in them.
>>
>> Anytime you are worried about things like this throw in a nodetool
>> repair. If you are using QUOURM for read and writes you will still be
>> getting consistent data, so long as you have only added one node. Or one
>> node every RF'th nodes.
>>
>> Cheers
>>
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 22/02/2013, at 9:55 PM, Jean-Armel Luce <ja...@gmail.com> wrote:
>>
>> Hi Aaron,
>>
>> Thanks for your answer.
>>
>>
>> I apologize, I did a mistake in my 1st mail. The cluster was only 12
>> nodes instead of 16 (it is a test cluster).
>> There are 2 datacenters b1 and s1.
>>
>> Here is the result of nodetool status after adding a new node in the 1st
>> datacenter (dc s1):
>> root@node007:~# nodetool status
>> Datacenter: b1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN 10.234.72.135 10.71 GB 256 44.6%
>> 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
>> UN 10.234.72.134 16.74 GB 256 63.7%
>> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
>> UN 10.234.72.139 17.09 GB 256 62.0%
>> 95661392-ccd8-4592-a76f-1c99f7cdf23a e07
>> UN 10.234.72.138 10.96 GB 256 42.9%
>> 0d6725f0-1357-423d-85c1-153fb94257d5 e03
>> UN 10.234.72.137 11.09 GB 256 45.7%
>> 492190d7-3055-4167-8699-9c6560e28164 e03
>> UN 10.234.72.136 11.91 GB 256 41.1%
>> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
>> Datacenter: s1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN 10.98.255.139 16.94 GB 256 43.8%
>> 3523e80c-8468-4502-b334-79eabc3357f0 g10
>> UN 10.98.255.138 12.62 GB 256 42.4%
>> a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
>> UN 10.98.255.137 10.59 GB 256 38.4%
>> f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
>> UN 10.98.255.136 11.89 GB 256 42.9%
>> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
>> UN 10.98.255.135 10.29 GB 256 40.4%
>> e2d020a5-97a9-48d4-870c-d10b59858763 a09
>> UN 10.98.255.134 16.19 GB 256 52.3%
>> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
>> UN 10.98.255.140 127.84 KB 256 39.9%
>> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
>>
>> We can see that the new node (10.98.255.140) contains only 127,84KB.
>> We saw also that there was no network traffic between the nodes.
>>
>> Then we added a new node in the 2nd datacenter (dc b1)
>>
>>
>>
>> root@node007:~# nodetool status
>> Datacenter: b1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN 10.234.72.135 12.95 GB 256 42.0%
>> 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
>> UN 10.234.72.134 20.11 GB 256 53.1%
>> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
>> UN 10.234.72.140 122.25 KB 256 41.9%
>> 501ea498-8fed-4cc8-a23a-c99492bc4f26 e07
>> UN 10.234.72.139 20.46 GB 256 40.2%
>> 95661392-ccd8-4592-a76f-1c99f7cdf23a e07
>> UN 10.234.72.138 13.21 GB 256 40.9%
>> 0d6725f0-1357-423d-85c1-153fb94257d5 e03
>> UN 10.234.72.137 13.34 GB 256 42.9%
>> 492190d7-3055-4167-8699-9c6560e28164 e03
>> UN 10.234.72.136 14.16 GB 256 39.0%
>> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
>> Datacenter: s1
>> ==============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN 10.98.255.139 19.19 GB 256 43.8%
>> 3523e80c-8468-4502-b334-79eabc3357f0 g10
>> UN 10.98.255.138 14.9 GB 256 42.4%
>> a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
>> UN 10.98.255.137 12.49 GB 256 38.4%
>> f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
>> UN 10.98.255.136 14.13 GB 256 42.9%
>> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
>> UN 10.98.255.135 12.16 GB 256 40.4%
>> e2d020a5-97a9-48d4-870c-d10b59858763 a09
>> UN 10.98.255.134 18.85 GB 256 52.3%
>> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
>> UN 10.98.255.140 2.24 GB 256 39.9%
>> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
>>
>>
>> We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
>> The new node in the 1st datacenter contains now 2,24 GB because we were
>> inserting data in the cluster while adding the new nodes.
>>
>> Then we started a repair from the new node in the 2nd datacenter :
>> time nodetool repair
>>
>>
>> We can see that the old nodes are sending data to the new node :
>> root@node007:~# nodetool netstats
>> Mode: NORMAL
>> Not sending any streams.
>> Streaming from: /10.98.255.137
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
>> sections=130 progress=0/15598366 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
>> sections=107 progress=0/429517 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
>> sections=109 progress=0/696057 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
>> sections=57 progress=0/189844 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
>> sections=124 progress=56492032/4597955 - 1228%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
>> sections=113 progress=0/850632 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
>> sections=130 progress=0/16815423 - 0%
>> Streaming from: /10.98.255.136
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
>> sections=104 progress=0/3545520 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
>> sections=43 progress=0/252976 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
>> sections=91 progress=0/443365 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
>> sections=101 progress=11337728/4188624 - 270%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
>> sections=94 progress=0/758418 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
>> sections=88 progress=0/396552 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
>> sections=105 progress=0/12363907 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
>> sections=87 progress=0/506537 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
>> sections=42 progress=0/197531 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
>> sections=110 progress=0/12689898 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
>> sections=83 progress=0/379628 - 0%
>> Streaming from: /10.98.255.134
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
>> sections=108 progress=0/10715348 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
>> sections=82 progress=0/8177161 - 0%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
>> sections=84 progress=25613959168/8556942 - 299335%
>> hbxtest:
>> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
>> sections=79 progress=0/506293 - 0%
>> Pool Name Active Pending Completed
>> Commands n/a 0 835
>> Responses n/a 0 6765
>>
>> So, it looks that the repair is required if we want to add new nodes in
>> our platform, but I don't understand why.
>>
>> Regards.
>>
>> Jean Armel
>>
>>
>>
>>
>
Re: Adding new nodes in a cluster with virtual nodes
Posted by Jean-Armel Luce <ja...@gmail.com>.
Thanks Aaron.
I shall investigate more next week about this.
regards.
Jean Armel
2013/2/22 aaron morton <aa...@thelastpickle.com>
> So, it looks that the repair is required if we want to add new nodes in
> our platform, but I don't understand why.
>
> Bootstrapping should take care of it. But new seed nodes do not bootstrap.
> Check the logs on the nodes you added to see what messages have "bootstrap"
> in them.
>
> Anytime you are worried about things like this throw in a nodetool repair.
> If you are using QUOURM for read and writes you will still be getting
> consistent data, so long as you have only added one node. Or one node every
> RF'th nodes.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/02/2013, at 9:55 PM, Jean-Armel Luce <ja...@gmail.com> wrote:
>
> Hi Aaron,
>
> Thanks for your answer.
>
>
> I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes
> instead of 16 (it is a test cluster).
> There are 2 datacenters b1 and s1.
>
> Here is the result of nodetool status after adding a new node in the 1st
> datacenter (dc s1):
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 10.234.72.135 10.71 GB 256 44.6%
> 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
> UN 10.234.72.134 16.74 GB 256 63.7%
> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
> UN 10.234.72.139 17.09 GB 256 62.0%
> 95661392-ccd8-4592-a76f-1c99f7cdf23a e07
> UN 10.234.72.138 10.96 GB 256 42.9%
> 0d6725f0-1357-423d-85c1-153fb94257d5 e03
> UN 10.234.72.137 11.09 GB 256 45.7%
> 492190d7-3055-4167-8699-9c6560e28164 e03
> UN 10.234.72.136 11.91 GB 256 41.1%
> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 10.98.255.139 16.94 GB 256 43.8%
> 3523e80c-8468-4502-b334-79eabc3357f0 g10
> UN 10.98.255.138 12.62 GB 256 42.4%
> a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
> UN 10.98.255.137 10.59 GB 256 38.4%
> f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
> UN 10.98.255.136 11.89 GB 256 42.9%
> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
> UN 10.98.255.135 10.29 GB 256 40.4%
> e2d020a5-97a9-48d4-870c-d10b59858763 a09
> UN 10.98.255.134 16.19 GB 256 52.3%
> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
> UN 10.98.255.140 127.84 KB 256 39.9%
> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
>
> We can see that the new node (10.98.255.140) contains only 127,84KB.
> We saw also that there was no network traffic between the nodes.
>
> Then we added a new node in the 2nd datacenter (dc b1)
>
>
>
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 10.234.72.135 12.95 GB 256 42.0%
> 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
> UN 10.234.72.134 20.11 GB 256 53.1%
> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
> UN 10.234.72.140 122.25 KB 256 41.9%
> 501ea498-8fed-4cc8-a23a-c99492bc4f26 e07
> UN 10.234.72.139 20.46 GB 256 40.2%
> 95661392-ccd8-4592-a76f-1c99f7cdf23a e07
> UN 10.234.72.138 13.21 GB 256 40.9%
> 0d6725f0-1357-423d-85c1-153fb94257d5 e03
> UN 10.234.72.137 13.34 GB 256 42.9%
> 492190d7-3055-4167-8699-9c6560e28164 e03
> UN 10.234.72.136 14.16 GB 256 39.0%
> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 10.98.255.139 19.19 GB 256 43.8%
> 3523e80c-8468-4502-b334-79eabc3357f0 g10
> UN 10.98.255.138 14.9 GB 256 42.4%
> a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
> UN 10.98.255.137 12.49 GB 256 38.4%
> f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
> UN 10.98.255.136 14.13 GB 256 42.9%
> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
> UN 10.98.255.135 12.16 GB 256 40.4%
> e2d020a5-97a9-48d4-870c-d10b59858763 a09
> UN 10.98.255.134 18.85 GB 256 52.3%
> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
> UN 10.98.255.140 2.24 GB 256 39.9%
> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
>
>
> We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
> The new node in the 1st datacenter contains now 2,24 GB because we were
> inserting data in the cluster while adding the new nodes.
>
> Then we started a repair from the new node in the 2nd datacenter :
> time nodetool repair
>
>
> We can see that the old nodes are sending data to the new node :
> root@node007:~# nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Streaming from: /10.98.255.137
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
> sections=130 progress=0/15598366 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
> sections=107 progress=0/429517 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
> sections=109 progress=0/696057 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
> sections=57 progress=0/189844 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
> sections=124 progress=56492032/4597955 - 1228%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
> sections=113 progress=0/850632 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
> sections=130 progress=0/16815423 - 0%
> Streaming from: /10.98.255.136
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
> sections=104 progress=0/3545520 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
> sections=43 progress=0/252976 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
> sections=91 progress=0/443365 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
> sections=101 progress=11337728/4188624 - 270%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
> sections=94 progress=0/758418 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
> sections=88 progress=0/396552 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
> sections=105 progress=0/12363907 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
> sections=87 progress=0/506537 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
> sections=42 progress=0/197531 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
> sections=110 progress=0/12689898 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
> sections=83 progress=0/379628 - 0%
> Streaming from: /10.98.255.134
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
> sections=108 progress=0/10715348 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
> sections=82 progress=0/8177161 - 0%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
> sections=84 progress=25613959168/8556942 - 299335%
> hbxtest:
> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
> sections=79 progress=0/506293 - 0%
> Pool Name Active Pending Completed
> Commands n/a 0 835
> Responses n/a 0 6765
>
> So, it looks that the repair is required if we want to add new nodes in
> our platform, but I don't understand why.
>
> Regards.
>
> Jean Armel
>
>
>
>
Re: Adding new nodes in a cluster with virtual nodes
Posted by aaron morton <aa...@thelastpickle.com>.
> So, it looks that the repair is required if we want to add new nodes in our platform, but I don't understand why.
Bootstrapping should take care of it. But new seed nodes do not bootstrap. Check the logs on the nodes you added to see what messages have "bootstrap" in them.
Anytime you are worried about things like this throw in a nodetool repair. If you are using QUOURM for read and writes you will still be getting consistent data, so long as you have only added one node. Or one node every RF'th nodes.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 22/02/2013, at 9:55 PM, Jean-Armel Luce <ja...@gmail.com> wrote:
> Hi Aaron,
>
> Thanks for your answer.
>
>
> I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes instead of 16 (it is a test cluster).
> There are 2 datacenters b1 and s1.
>
> Here is the result of nodetool status after adding a new node in the 1st datacenter (dc s1):
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.234.72.135 10.71 GB 256 44.6% 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
> UN 10.234.72.134 16.74 GB 256 63.7% f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
> UN 10.234.72.139 17.09 GB 256 62.0% 95661392-ccd8-4592-a76f-1c99f7cdf23a e07
> UN 10.234.72.138 10.96 GB 256 42.9% 0d6725f0-1357-423d-85c1-153fb94257d5 e03
> UN 10.234.72.137 11.09 GB 256 45.7% 492190d7-3055-4167-8699-9c6560e28164 e03
> UN 10.234.72.136 11.91 GB 256 41.1% 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.98.255.139 16.94 GB 256 43.8% 3523e80c-8468-4502-b334-79eabc3357f0 g10
> UN 10.98.255.138 12.62 GB 256 42.4% a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
> UN 10.98.255.137 10.59 GB 256 38.4% f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
> UN 10.98.255.136 11.89 GB 256 42.9% 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
> UN 10.98.255.135 10.29 GB 256 40.4% e2d020a5-97a9-48d4-870c-d10b59858763 a09
> UN 10.98.255.134 16.19 GB 256 52.3% 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
> UN 10.98.255.140 127.84 KB 256 39.9% 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
>
> We can see that the new node (10.98.255.140) contains only 127,84KB.
> We saw also that there was no network traffic between the nodes.
>
> Then we added a new node in the 2nd datacenter (dc b1)
>
>
>
> root@node007:~# nodetool status
> Datacenter: b1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.234.72.135 12.95 GB 256 42.0% 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
> UN 10.234.72.134 20.11 GB 256 53.1% f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
> UN 10.234.72.140 122.25 KB 256 41.9% 501ea498-8fed-4cc8-a23a-c99492bc4f26 e07
> UN 10.234.72.139 20.46 GB 256 40.2% 95661392-ccd8-4592-a76f-1c99f7cdf23a e07
> UN 10.234.72.138 13.21 GB 256 40.9% 0d6725f0-1357-423d-85c1-153fb94257d5 e03
> UN 10.234.72.137 13.34 GB 256 42.9% 492190d7-3055-4167-8699-9c6560e28164 e03
> UN 10.234.72.136 14.16 GB 256 39.0% 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
> Datacenter: s1
> ==============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.98.255.139 19.19 GB 256 43.8% 3523e80c-8468-4502-b334-79eabc3357f0 g10
> UN 10.98.255.138 14.9 GB 256 42.4% a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
> UN 10.98.255.137 12.49 GB 256 38.4% f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
> UN 10.98.255.136 14.13 GB 256 42.9% 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
> UN 10.98.255.135 12.16 GB 256 40.4% e2d020a5-97a9-48d4-870c-d10b59858763 a09
> UN 10.98.255.134 18.85 GB 256 52.3% 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
> UN 10.98.255.140 2.24 GB 256 39.9% 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
>
>
> We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
> The new node in the 1st datacenter contains now 2,24 GB because we were inserting data in the cluster while adding the new nodes.
>
> Then we started a repair from the new node in the 2nd datacenter :
> time nodetool repair
>
>
> We can see that the old nodes are sending data to the new node :
> root@node007:~# nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Streaming from: /10.98.255.137
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db sections=130 progress=0/15598366 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db sections=107 progress=0/429517 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db sections=109 progress=0/696057 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db sections=57 progress=0/189844 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db sections=124 progress=56492032/4597955 - 1228%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db sections=113 progress=0/850632 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db sections=130 progress=0/16815423 - 0%
> Streaming from: /10.98.255.136
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db sections=104 progress=0/3545520 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db sections=43 progress=0/252976 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db sections=91 progress=0/443365 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db sections=101 progress=11337728/4188624 - 270%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db sections=94 progress=0/758418 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db sections=88 progress=0/396552 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db sections=105 progress=0/12363907 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db sections=87 progress=0/506537 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db sections=42 progress=0/197531 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db sections=110 progress=0/12689898 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db sections=83 progress=0/379628 - 0%
> Streaming from: /10.98.255.134
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db sections=108 progress=0/10715348 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db sections=82 progress=0/8177161 - 0%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db sections=84 progress=25613959168/8556942 - 299335%
> hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db sections=79 progress=0/506293 - 0%
> Pool Name Active Pending Completed
> Commands n/a 0 835
> Responses n/a 0 6765
>
> So, it looks that the repair is required if we want to add new nodes in our platform, but I don't understand why.
>
> Regards.
>
> Jean Armel
>
>
Re: Adding new nodes in a cluster with virtual nodes
Posted by Jean-Armel Luce <ja...@gmail.com>.
Hi Aaron,
Thanks for your answer.
I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes
instead of 16 (it is a test cluster).
There are 2 datacenters b1 and s1.
Here is the result of nodetool status after adding a new node in the 1st
datacenter (dc s1):
root@node007:~# nodetool status
Datacenter: b1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.234.72.135 10.71 GB 256 44.6%
2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
UN 10.234.72.134 16.74 GB 256 63.7%
f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
UN 10.234.72.139 17.09 GB 256 62.0%
95661392-ccd8-4592-a76f-1c99f7cdf23a e07
UN 10.234.72.138 10.96 GB 256 42.9%
0d6725f0-1357-423d-85c1-153fb94257d5 e03
UN 10.234.72.137 11.09 GB 256 45.7%
492190d7-3055-4167-8699-9c6560e28164 e03
UN 10.234.72.136 11.91 GB 256 41.1%
3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
Datacenter: s1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.98.255.139 16.94 GB 256 43.8%
3523e80c-8468-4502-b334-79eabc3357f0 g10
UN 10.98.255.138 12.62 GB 256 42.4%
a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
UN 10.98.255.137 10.59 GB 256 38.4%
f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
UN 10.98.255.136 11.89 GB 256 42.9%
36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
UN 10.98.255.135 10.29 GB 256 40.4%
e2d020a5-97a9-48d4-870c-d10b59858763 a09
UN 10.98.255.134 16.19 GB 256 52.3%
73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
UN 10.98.255.140 127.84 KB 256 39.9%
3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
We can see that the new node (10.98.255.140) contains only 127,84KB.
We saw also that there was no network traffic between the nodes.
Then we added a new node in the 2nd datacenter (dc b1)
root@node007:~# nodetool status
Datacenter: b1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.234.72.135 12.95 GB 256 42.0%
2fc583b2-822f-4347-9fab-5e9d10d548c9 c01
UN 10.234.72.134 20.11 GB 256 53.1%
f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01
UN 10.234.72.140 122.25 KB 256 41.9%
501ea498-8fed-4cc8-a23a-c99492bc4f26 e07
UN 10.234.72.139 20.46 GB 256 40.2%
95661392-ccd8-4592-a76f-1c99f7cdf23a e07
UN 10.234.72.138 13.21 GB 256 40.9%
0d6725f0-1357-423d-85c1-153fb94257d5 e03
UN 10.234.72.137 13.34 GB 256 42.9%
492190d7-3055-4167-8699-9c6560e28164 e03
UN 10.234.72.136 14.16 GB 256 39.0%
3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01
Datacenter: s1
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 10.98.255.139 19.19 GB 256 43.8%
3523e80c-8468-4502-b334-79eabc3357f0 g10
UN 10.98.255.138 14.9 GB 256 42.4%
a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02
UN 10.98.255.137 12.49 GB 256 38.4%
f851b6ee-f1e4-431b-8beb-e7b173a77342 i02
UN 10.98.255.136 14.13 GB 256 42.9%
36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09
UN 10.98.255.135 12.16 GB 256 40.4%
e2d020a5-97a9-48d4-870c-d10b59858763 a09
UN 10.98.255.134 18.85 GB 256 52.3%
73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06
UN 10.98.255.140 2.24 GB 256 39.9%
3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10
We can see that the 2nd new node (10.234.72.140) contains only 122,25KB.
The new node in the 1st datacenter contains now 2,24 GB because we were
inserting data in the cluster while adding the new nodes.
Then we started a repair from the new node in the 2nd datacenter :
time nodetool repair
We can see that the old nodes are sending data to the new node :
root@node007:~# nodetool netstats
Mode: NORMAL
Not sending any streams.
Streaming from: /10.98.255.137
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db
sections=130 progress=0/15598366 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db
sections=107 progress=0/429517 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=109 progress=0/696057 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db
sections=57 progress=0/189844 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db
sections=124 progress=56492032/4597955 - 1228%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db
sections=113 progress=0/850632 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db
sections=130 progress=0/16815423 - 0%
Streaming from: /10.98.255.136
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db
sections=104 progress=0/3545520 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=43 progress=0/252976 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=91 progress=0/443365 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db
sections=101 progress=11337728/4188624 - 270%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db
sections=94 progress=0/758418 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db
sections=88 progress=0/396552 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db
sections=105 progress=0/12363907 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=87 progress=0/506537 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db
sections=42 progress=0/197531 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db
sections=110 progress=0/12689898 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=83 progress=0/379628 - 0%
Streaming from: /10.98.255.134
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db
sections=108 progress=0/10715348 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db
sections=82 progress=0/8177161 - 0%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db
sections=84 progress=25613959168/8556942 - 299335%
hbxtest:
/var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db
sections=79 progress=0/506293 - 0%
Pool Name Active Pending Completed
Commands n/a 0 835
Responses n/a 0 6765
So, it looks that the repair is required if we want to add new nodes in our
platform, but I don't understand why.
Regards.
Jean Armel
Re: Adding new nodes in a cluster with virtual nodes
Posted by aaron morton <aa...@thelastpickle.com>.
> After the bootstrap is finished in the new node, no data has been moved automatically from the old nodes to this new node.
What did nodetool ring say ?
> Please, could you tell me if we need to perform a nodetool repair after the bootstrap of the new node ?
You should not, bootstrap brings the node up with the data it should have.
If you have any doubt though, running nodetool repair -pr on the new node cannot hurt.
> What happens if we perform a nodetool cleanup in the old nodes before doing the nodetool repair ? (Is there a risk of loosing some data ?)
Potentially yes.
The simple thing would be to wait until a full repair cycle (on all nodes) to complete.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 22/02/2013, at 3:44 AM, Jean-Armel Luce <ja...@gmail.com> wrote:
> Hello,
>
> We are using Cassandra 1.2.0.
> We have a cluster of 16 physical nodes, each node has 256 virtual nodes.
> We want to add 2 new nodes in our cluster : we follow the procedure as explained here : http://www.datastax.com/docs/1.2/operations/add_replace_nodes.
>
> After starting 1 of the new node, we can see that this new node has 256 tokens ==>looks good
> We can see that this node is in the ring (using nodetool status) ==> looks good
> After the bootstrap is finished in the new node, no data has been moved automatically from the old nodes to this new node.
>
>
> However, when we send insert queries in our cluster, the new node accepts to insert the new rows.
>
> Please, could you tell me if we need to perform a nodetool repair after the bootstrap of the new node ?
> What happens if we perform a nodetool cleanup in the old nodes before doing the nodetool repair ? (Is there a risk of loosing some data ?)
>
> Regards.
>
> Jean Armel