You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Henry Luo <hl...@choicestream.com> on 2010/11/01 23:01:38 UTC
Adding nodes in new data center
We have a cluster running in one data center, and are adding some in a new data center. There are some data already in the current cluster.
We did something wrong at first by not having AutoBootstrap on, when we saw no data being migrated. So we wiped out the data on the new nodes, and then start them up with AutoBootstrap on. We still did not see data migrated.
I'm thinking that we need to do a nodetool decommission the new nodes, and start them with AutoBootstrap on. Does that make sense?
Thanks.
Henry
________________________________
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.
Streaming still a problem in 0.6.6?
Posted by Henry Luo <hl...@choicestream.com>.
We are adding node into a cluster with some data. The new node is in Bootstrapping mode for a long time, waiting for data to be streamed to it, but the streaming seems to get stuck.
Mode: Normal
Streaming to: /10.110.10.10
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-4277-Data.db 6360157440/6360157440
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-20728-Index.db 0/2132869987
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-20728-Filter.db 0/103873765
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-20728-Data.db 0/25959995677
Not receiving any streams.
node 2
Mode: Normal
Streaming to: /10.110.10.10
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3965-Data.db 6366356966/6366356966
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-19063-Index.db 0/2147861349
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-19063-Filter.db 0/107220325
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-19063-Data.db 0/26026489399
Not receiving any streams.
node 3
Mode: Normal
Streaming to: /10.110.10.10
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3916-Index.db 0/2762751688
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3916-Filter.db 0/40930525
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3916-Data.db 0/19091595776
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-18821-Index.db 0/6428352175
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-18821-Filter.db 0/97584205
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-18821-Data.db 0/77971715253
[cassandra@sea-cass1 ~]$ /opt/choicestream/deps/apache-cassandra-0.6.6/bin/nodetool -h localhost streams
Mode: Bootstrapping
Not sending any streams.
Streaming from: /10.100.10.68
Profile: /opt/choicestream/data/cassandra/data/Profile/U_Profiles-tmp-3-Index.db 0/2762751688
Profile: /opt/choicestream/data/cassandra/data/Profile/U_Profiles-tmp-3-Filter.db 0/40930525
Profile: /opt/choicestream/data/cassandra/data/Profile/U_Profiles-tmp-3-Data.db 0/19091595776
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-3-Index.db 0/6428352175
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-3-Filter.db 0/97584205
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-3-Data.db 0/77971715253
Streaming from: /10.100.10.64
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-2-Index.db 0/2132869987
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-2-Filter.db 0/103873765
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-2-Data.db 0/25959995677
Streaming from: /10.100.10.66
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-1-Index.db 0/2147861349
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-1-Filter.db 0/107220325
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-1-Data.db 0/26026489399
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.
RE: Adding nodes in new data center
Posted by Henry Luo <hl...@choicestream.com>.
We tried it again with one node from the new data center, and this time, it got stuck in streaming. We had this problem while on 0.6.1, now we are on 0.6.6.
It appears that some streaming finished, but somehow the source nodes are waiting for something.
Anyone has a clue?
Thanks.
Mode: Normal
Streaming to: /10.110.10.10
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-4277-Data.db 6360157440/6360157440
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-20728-Index.db 0/2132869987
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-20728-Filter.db 0/103873765
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-20728-Data.db 0/25959995677
Not receiving any streams.
node 2
Mode: Normal
Streaming to: /10.110.10.10
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3965-Data.db 6366356966/6366356966
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-19063-Index.db 0/2147861349
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-19063-Filter.db 0/107220325
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-19063-Data.db 0/26026489399
Not receiving any streams.
node 3
Mode: Normal
Streaming to: /10.110.10.10
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3916-Index.db 0/2762751688
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3916-Filter.db 0/40930525
/opt/choicestream/data/cassandra/data/Profile/stream/U_Profiles-3916-Data.db 0/19091595776
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-18821-Index.db 0/6428352175
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-18821-Filter.db 0/97584205
/opt/choicestream/data/cassandra/data/Profile/stream/T_Profiles-18821-Data.db 0/77971715253
[cassandra@sea-cass1 ~]$ /opt/choicestream/deps/apache-cassandra-0.6.6/bin/nodetool -h localhost streams
Mode: Bootstrapping
Not sending any streams.
Streaming from: /10.100.10.68
Profile: /opt/choicestream/data/cassandra/data/Profile/U_Profiles-tmp-3-Index.db 0/2762751688
Profile: /opt/choicestream/data/cassandra/data/Profile/U_Profiles-tmp-3-Filter.db 0/40930525
Profile: /opt/choicestream/data/cassandra/data/Profile/U_Profiles-tmp-3-Data.db 0/19091595776
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-3-Index.db 0/6428352175
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-3-Filter.db 0/97584205
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-3-Data.db 0/77971715253
Streaming from: /10.100.10.64
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-2-Index.db 0/2132869987
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-2-Filter.db 0/103873765
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-2-Data.db 0/25959995677
Streaming from: /10.100.10.66
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-1-Index.db 0/2147861349
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-1-Filter.db 0/107220325
Profile: /opt/choicestream/data/cassandra/data/Profile/T_Profiles-tmp-1-Data.db 0/26026489399
-----Original Message-----
From: Henry Luo
Sent: Tuesday, November 02, 2010 12:10 PM
To: user@cassandra.apache.org
Subject: RE: Adding nodes in new data center
As suggested, we did do removetoken, set autobootstrap true, initial token and start. Now the new nodes are in the Boostrapping state for a while, but nothing seem to be happening at all.
Any idea?
Thanks.
Henry
-----Original Message-----
From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: Monday, November 01, 2010 6:08 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes in new data center
On Mon, Nov 1, 2010 at 6:01 PM, Henry Luo <hl...@choicestream.com> wrote:
> We have a cluster running in one data center, and are adding some in a new
> data center. There are some data already in the current cluster.
>
> We did something wrong at first by not having AutoBootstrap on, when we saw
> no data being migrated. So we wiped out the data on the new nodes, and then
> start them up with AutoBootstrap on. We still did not see data migrated.
>
>
>
> I'm thinking that we need to do a nodetool decommission the new nodes, and
> start them with AutoBootstrap on. Does that make sense?
>
>
>
> Thanks.
>
> Henry
>
> ________________________________
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential, proprietary, and/or
> privileged material. Any review, retransmission, dissemination or other use
> of, or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from all
> computers.
>
Correct. Once you have started a node with auto-bootstrap=false,
changing the configuration file no longer works. The nodes are
bootstrapped and joined to the cluster.
Yes. you want to decommission or removetoken the new nodes. Set
auto-bootstrap to true, set an initialtoken and start them.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.
RE: Adding nodes in new data center
Posted by Henry Luo <hl...@choicestream.com>.
As suggested, we did do removetoken, set autobootstrap true, initial token and start. Now the new nodes are in the Boostrapping state for a while, but nothing seem to be happening at all.
Any idea?
Thanks.
Henry
-----Original Message-----
From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: Monday, November 01, 2010 6:08 PM
To: user@cassandra.apache.org
Subject: Re: Adding nodes in new data center
On Mon, Nov 1, 2010 at 6:01 PM, Henry Luo <hl...@choicestream.com> wrote:
> We have a cluster running in one data center, and are adding some in a new
> data center. There are some data already in the current cluster.
>
> We did something wrong at first by not having AutoBootstrap on, when we saw
> no data being migrated. So we wiped out the data on the new nodes, and then
> start them up with AutoBootstrap on. We still did not see data migrated.
>
>
>
> I'm thinking that we need to do a nodetool decommission the new nodes, and
> start them with AutoBootstrap on. Does that make sense?
>
>
>
> Thanks.
>
> Henry
>
> ________________________________
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential, proprietary, and/or
> privileged material. Any review, retransmission, dissemination or other use
> of, or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from all
> computers.
>
Correct. Once you have started a node with auto-bootstrap=false,
changing the configuration file no longer works. The nodes are
bootstrapped and joined to the cluster.
Yes. you want to decommission or removetoken the new nodes. Set
auto-bootstrap to true, set an initialtoken and start them.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.
Re: Adding nodes in new data center
Posted by Edward Capriolo <ed...@gmail.com>.
On Mon, Nov 1, 2010 at 6:01 PM, Henry Luo <hl...@choicestream.com> wrote:
> We have a cluster running in one data center, and are adding some in a new
> data center. There are some data already in the current cluster.
>
> We did something wrong at first by not having AutoBootstrap on, when we saw
> no data being migrated. So we wiped out the data on the new nodes, and then
> start them up with AutoBootstrap on. We still did not see data migrated.
>
>
>
> I’m thinking that we need to do a nodetool decommission the new nodes, and
> start them with AutoBootstrap on. Does that make sense?
>
>
>
> Thanks.
>
> Henry
>
> ________________________________
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential, proprietary, and/or
> privileged material. Any review, retransmission, dissemination or other use
> of, or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from all
> computers.
>
Correct. Once you have started a node with auto-bootstrap=false,
changing the configuration file no longer works. The nodes are
bootstrapped and joined to the cluster.
Yes. you want to decommission or removetoken the new nodes. Set
auto-bootstrap to true, set an initialtoken and start them.