You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "XL.Pan" <pa...@sina.com> on 2010/01/14 13:30:03 UTC
replace a bad node through bootstrapping
Hi ALL:
*My issues:
I have a few high-capacity servers, which has about 5T disk space. I know there are 2 solutions for handling failure, through the wiki. But it's not very conveninte for me. That is :
Solution 1: new node + removetoken
Add a new node will make lots of data transfer between machines. And, of couse, the removetoken operation will make the data be transferred too. That means the data will be transferred between machines twice.
Solution 2: repaire
Though I have not try this operation yes, I find it is a heavy operation, because it will trigger the major compaction.
*My objects:
I think the most conveninte way to handle one machine�s failure is that :
"1) Replace the bad node with a new one; 2) Copy data from the other nodes to the new node. 3) Start the service on the new node."
*Why not the standard boostrap?
At the first time, I think the boostrap can deal with this. But I�m wrong, because of 3 problems.
1) The new node, which has no data before bootstrapping, will be found by other nodes as soon as the Gossiper is OK. Before it is changed to boostrapping mode, it will receive many routed message and of couse, all of them failed to response.
2) After it is changed to booststrapping mode, other nodes will remove it from the ring. That means the architecture of the ring has been changed while the replication count is not restored. As a result, many read requests are routed to the wrong nodes.
3) The new node, which is in the bootstrapping mode, think itself is a new one in the ring and the replication count in the ring have been restored, so it calculates the replication nodes without itself in the ring. Of couse, it�s not right.
*My solutions:
1) Hide the new node�s state from the other nodes until the bootstrapping has done.
Then, the new node can stream the data and will not worry other nodes route messages at that time, because other ones think it�s down.
It seems ok to take off itself from the �deltaEpStateMap� in the ACK and ACK2. As a result, other nodes will not see it until put itself back.
2) Add itselft into the ring temporarily while calculating the replication nodes.
In this way, the new node will see the original ring and can find the right replication nodes.
*My Questions:
Am I right? What�s the problem in my design? Why not before?
--------------
XL.Pan
2010-01-14
Re: Re: replace a bad node through bootstrapping
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
<ma...@gmail.com> wrote:
> If a node's data has been damaged, you cannot use new node replace old one directly, unless 'removetoken' first.
>
> But, (suppose node A is dead)
> 'removetoken' will complement missing replica due A's death first, it will generate lot data on other nodes, say it's B, C, D
> After add new node and copy data from other node through bootstrapping, you have to 'cleanup' data just
> generate from ' removetoken ' on B, C, D
>
> So, B/C/D will have heavy I/O load (half of them is waste) due to repair A, in pan's case, it will be 5TB (and will cause days...)
>
> Pan try to invent a method to repair A directly through streaming, and have less impact on other nodes.
Thanks for clarifying that.
I thought we agreed in your last thread about this that bootstrapping
a replacement node (the old node can be the replacement, as long as
you change its IP address) first, then removing the entry for the dead
one, would be a reasonable procedure here.
-Jonathan
RE: Re: replace a bad node through bootstrapping
Posted by Michael Lee <ma...@gmail.com>.
Sorry, My bad!
So, if use same token different IP, ' nodeprobe removetoken ' process is not
needed, am I right?
If so, I think the wiki page should state it clear, my advice....
And once again, the replaced node damage again, we replace the bad one with
new node, this time we use the same token again, and use the 'old' IP
For example:
Step1: replace A (token=11111, ip=1.1.1.1) with B(token=11111, ip=2.2.2.2),
according we discussed before, it will work
Step2: we fix A, format the whole file system, now B go to bad, we replace
B(token=11111, ip=2.2.2.2) with A (token=11111, ip=1.1.1.1) because we only
one backup IP and backup node.
Does step 2 still work？
---------END----------
-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Tuesday, January 19, 2010 9:57 AM
To: cassandra-user@incubator.apache.org
Subject: Re: Re: replace a bad node through bootstrapping
This is described in the "Handling failure" section of the Operations page.
I believe it will work even if you use the same token as the old node, yes.
-Jonathan
2010/1/18 Michael Lee <ma...@gmail.com>:
> Even good node has the same token as bad one?
>
> Is it an 'documented' operation? I haven't seen it in wiki
links(Operation).
>
> ---------END----------
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Monday, January 18, 2010 11:22 PM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Re: Re: replace a bad node through bootstrapping
>
> yes
>
> On Mon, Jan 18, 2010 at 4:37 AM, XL.Pan <pa...@sina.com> wrote:
>> Hi Jonathan:
>> "the old node can be the replacement, as long as you change its IP
> address"
>>
>> Do you mean that the operations to replace a bad node is :
>> 1) choose a new machine which has the same configuration, eg.
> InitialToken, and has a different IP address;
>> 2) start the new machine, which will start boostrapping;
>> 3) After bootstrapping, the new machine will restore the data as before.
>>
>> (All nodes' InitialToken are set manully)
>>
>> I have tried in this way and that looks ok. Is this a good way? :-)
>> Thanks !!
>>
>>
>> ------------------
>> XL.Pan
>> 2010-01-18
>>
>> -------------------------------------------------------------
>> 发件人：Jonathan Ellis
>> 发送日期：2010-01-15 11:12:05
>> 收件人：cassandra-user
>> 抄送：
>> 主题：Re: Re: replace a bad node through bootstrapping
>>
>> On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
>> <ma...@gmail.com> wrote:
>>> If a node's data has been damaged, you cannot use new node replace old
> one directly, unless 'removetoken' first.
>>>
>>> But, (suppose node A is dead)
>>> 'removetoken' will complement missing replica due A's death first, it
> will generate lot data on other nodes, say it's B, C, D
>>> After add new node and copy data from other node through bootstrapping,
> you have to 'cleanup' data just
>>> generate from ' removetoken ' on B, C, D
>>>
>>> So, B/C/D will have heavy I/O load (half of them is waste) due to repair
> A, in pan's case, it will be 5TB (and will cause days...)
>>>
>>> Pan try to invent a method to repair A directly through streaming, and
> have less impact on other nodes.
>>
>> Thanks for clarifying that.
>>
>> I thought we agreed in your last thread about this that bootstrapping
>> a replacement node (the old node can be the replacement, as long as
>> you change its IP address) first, then removing the entry for the dead
>> one, would be a reasonable procedure here.
>>
>> -Jonathan
>>
>
>
Re: RE: Re: replace a bad node through bootstrapping - throw
exception
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like you want to bootstrap to a token close to but not the same
as the old one then. :)
2010/1/19 XL.Pan <pa...@sina.com>:
> Hi Jonathan :
> I find other nodes will throw exception when I use the same TokenID but different IP address for a new node while bootstrapping. My operations are that :
> 1) down nodeA which token is T, IP address is IPA;
> 2) add a new node B which token is also T but IP address is IPB;
> 3) After nodeA knows nodeB is bootstrapping, A throw exception:
>
> DEBUG [GMFD:2] 2010-01-19 22:22:36,667 StorageService.java (line 439) Node /10.81.37.52 state bootstrapping, token 136112946768375385385349842972707284580
> ERROR [GMFD:2] 2010-01-19 22:22:36,667 DebuggableThreadPoolExecutor.java (line 157) Error in ThreadPoolExecutor
> java.lang.RuntimeException: Bootstrap Token collision between /10.81.37.65 and /10.81.37.52 (token 136112946768375385385349842972707284580
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:136)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:456)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:419)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:692)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:657)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:611)
> at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(Gossiper.java:979)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-01-19 22:22:36,667 CassandraDaemon.java (line 71) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.RuntimeException: Bootstrap Token collision between /10.81.37.65 and /10.81.37.52 (token 136112946768375385385349842972707284580
> at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:136)
> at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:456)
> at org.apache.cassandra.service.StorageService.onChange(StorageService.java:419)
> at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:692)
> at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:657)
> at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:611)
> at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(Gossiper.java:979)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
> I traced the codes and find the reason is that :
> public void addBootstrapToken(Token token, InetAddress endpoint)
> {
> ......
> oldEndPoint = bootstrapTokens.get(token);
> if (oldEndPoint != null && !oldEndPoint.equals(endpoint))
> throw new RuntimeException("Bootstrap Token collision between " + oldEndPoint + " and " + endpoint + " (token " + token);
> // the exception is here --| :-)
> v
> oldEndPoint = tokenToEndPointMap.get(token);
> if (oldEndPoint != null && !oldEndPoint.equals(endpoint))
> throw new RuntimeException("Bootstrap Token collision between " + oldEndPoint + " and " + endpoint + " (token " + token);
> ......
> }
>
> In my enviroment, A is still in the tokenToEndPointMap though it's down. As a result, A will not know B is bootstrapping and will not route any request to B until B changes to NORMAL.
>
> I think the following changes may be one way to deal with this issue :
>
> if (oldEndPoint != null && !oldEndPoint.equals(endpoint) && FailureDetector.instance().isAlive(oldEndPoint))
> throw new RuntimeException("Bootstrap Token collision between " + oldEndPoint + " and " + endpoint + " (token " + token);
>
> Will you please give me some advice? Thanks!
>
>
> ------------------
> XL.Pan
> 2010-01-20
>
> -------------------------------------------------------------
> 发件人：Michael Lee
> 发送日期：2010-01-19 21:37:52
> 收件人：cassandra-user@incubator.apache.org
> 抄送：
> 主题：RE: Re: replace a bad node through bootstrapping
>
> Sorry, My bad!
>
> So, if use same token different IP, ' nodeprobe removetoken ' process is not
> needed, am I right?
>
> If so, I think the wiki page should state it clear, my advice....
>
> And once again, the replaced node damage again, we replace the bad one with
> new node, this time we use the same token again, and use the 'old' IP
> For example:
>
> Step1: replace A (token=11111, ip=1.1.1.1) with B(token=11111, ip=2.2.2.2),
> according we discussed before, it will work
> Step2: we fix A, format the whole file system, now B go to bad, we replace
> B(token=11111, ip=2.2.2.2) with A (token=11111, ip=1.1.1.1) because we only
> one backup IP and backup node.
>
> Does step 2 still work？
>
>
> ---------END----------
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Tuesday, January 19, 2010 9:57 AM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Re: replace a bad node through bootstrapping
>
> This is described in the "Handling failure" section of the Operations page.
>
> I believe it will work even if you use the same token as the old node, yes.
>
> -Jonathan
>
> 2010/1/18 Michael Lee <ma...@gmail.com>:
>> Even good node has the same token as bad one?
>>
>> Is it an 'documented' operation? I haven't seen it in wiki
> links(Operation).
>>
>> ---------END----------
>>
>> -----Original Message-----
>> From: Jonathan Ellis [mailto:jbellis@gmail.com]
>> Sent: Monday, January 18, 2010 11:22 PM
>> To: cassandra-user@incubator.apache.org
>> Subject: Re: Re: Re: replace a bad node through bootstrapping
>>
>> yes
>>
>> On Mon, Jan 18, 2010 at 4:37 AM, XL.Pan <pa...@sina.com> wrote:
>>> Hi Jonathan:
>>> "the old node can be the replacement, as long as you change its IP
>> address"
>>>
>>> Do you mean that the operations to replace a bad node is :
>>> 1) choose a new machine which has the same configuration, eg.
>> InitialToken, and has a different IP address;
>>> 2) start the new machine, which will start boostrapping;
>>> 3) After bootstrapping, the new machine will restore the data as before.
>>>
>>> (All nodes' InitialToken are set manully)
>>>
>>> I have tried in this way and that looks ok. Is this a good way? :-)
>>> Thanks !!
>>>
>>>
>>> ------------------
>>> XL.Pan
>>> 2010-01-18
>>>
>>> -------------------------------------------------------------
>>> 发件人：Jonathan Ellis
>>> 发送日期：2010-01-15 11:12:05
>>> 收件人：cassandra-user
>>> 抄送：
>>> 主题：Re: Re: replace a bad node through bootstrapping
>>>
>>> On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
>>> <ma...@gmail.com> wrote:
>>>> If a node's data has been damaged, you cannot use new node replace old
>> one directly, unless 'removetoken' first.
>>>>
>>>> But, (suppose node A is dead)
>>>> 'removetoken' will complement missing replica due A's death first, it
>> will generate lot data on other nodes, say it's B, C, D
>>>> After add new node and copy data from other node through bootstrapping,
>> you have to 'cleanup' data just
>>>> generate from ' removetoken ' on B, C, D
>>>>
>>>> So, B/C/D will have heavy I/O load (half of them is waste) due to repair
>> A, in pan's case, it will be 5TB (and will cause days...)
>>>>
>>>> Pan try to invent a method to repair A directly through streaming, and
>> have less impact on other nodes.
>>>
>>> Thanks for clarifying that.
>>>
>>> I thought we agreed in your last thread about this that bootstrapping
>>> a replacement node (the old node can be the replacement, as long as
>>> you change its IP address) first, then removing the entry for the dead
>>> one, would be a reasonable procedure here.
>>>
>>> -Jonathan
>>>
>>
>>
>
>
Re: RE: Re: replace a bad node through bootstrapping - throw exception
Posted by "XL.Pan" <pa...@sina.com>.
Hi Jonathan :
I find other nodes will throw exception when I use the same TokenID but different IP address for a new node while bootstrapping. My operations are that :
1) down nodeA which token is T, IP address is IPA;
2) add a new node B which token is also T but IP address is IPB;
3) After nodeA knows nodeB is bootstrapping, A throw exception:
DEBUG [GMFD:2] 2010-01-19 22:22:36,667 StorageService.java (line 439) Node /10.81.37.52 state bootstrapping, token 136112946768375385385349842972707284580
ERROR [GMFD:2] 2010-01-19 22:22:36,667 DebuggableThreadPoolExecutor.java (line 157) Error in ThreadPoolExecutor
java.lang.RuntimeException: Bootstrap Token collision between /10.81.37.65 and /10.81.37.52 (token 136112946768375385385349842972707284580
at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:136)
at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:456)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:419)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:692)
at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:657)
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:611)
at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(Gossiper.java:979)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ERROR [GMFD:2] 2010-01-19 22:22:36,667 CassandraDaemon.java (line 71) Fatal exception in thread Thread[GMFD:2,5,main]
java.lang.RuntimeException: Bootstrap Token collision between /10.81.37.65 and /10.81.37.52 (token 136112946768375385385349842972707284580
at org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:136)
at org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:456)
at org.apache.cassandra.service.StorageService.onChange(StorageService.java:419)
at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:692)
at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:657)
at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:611)
at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(Gossiper.java:979)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
I traced the codes and find the reason is that :
public void addBootstrapToken(Token token, InetAddress endpoint)
{
......
oldEndPoint = bootstrapTokens.get(token);
if (oldEndPoint != null && !oldEndPoint.equals(endpoint))
throw new RuntimeException("Bootstrap Token collision between " + oldEndPoint + " and " + endpoint + " (token " + token);
// the exception is here --| :-)
v
oldEndPoint = tokenToEndPointMap.get(token);
if (oldEndPoint != null && !oldEndPoint.equals(endpoint))
throw new RuntimeException("Bootstrap Token collision between " + oldEndPoint + " and " + endpoint + " (token " + token);
......
}
In my enviroment, A is still in the tokenToEndPointMap though it's down. As a result, A will not know B is bootstrapping and will not route any request to B until B changes to NORMAL.
I think the following changes may be one way to deal with this issue :
if (oldEndPoint != null && !oldEndPoint.equals(endpoint) && FailureDetector.instance().isAlive(oldEndPoint))
throw new RuntimeException("Bootstrap Token collision between " + oldEndPoint + " and " + endpoint + " (token " + token);
Will you please give me some advice? Thanks!
------------------
XL.Pan
2010-01-20
-------------------------------------------------------------
发件人：Michael Lee
发送日期：2010-01-19 21:37:52
收件人：cassandra-user@incubator.apache.org
抄送：
主题：RE: Re: replace a bad node through bootstrapping
Sorry, My bad!
So, if use same token different IP, ' nodeprobe removetoken ' process is not
needed, am I right?
If so, I think the wiki page should state it clear, my advice....
And once again, the replaced node damage again, we replace the bad one with
new node, this time we use the same token again, and use the 'old' IP
For example:
Step1: replace A (token=11111, ip=1.1.1.1) with B(token=11111, ip=2.2.2.2),
according we discussed before, it will work
Step2: we fix A, format the whole file system, now B go to bad, we replace
B(token=11111, ip=2.2.2.2) with A (token=11111, ip=1.1.1.1) because we only
one backup IP and backup node.
Does step 2 still work？
---------END----------
-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Tuesday, January 19, 2010 9:57 AM
To: cassandra-user@incubator.apache.org
Subject: Re: Re: replace a bad node through bootstrapping
This is described in the "Handling failure" section of the Operations page.
I believe it will work even if you use the same token as the old node, yes.
-Jonathan
2010/1/18 Michael Lee <ma...@gmail.com>:
> Even good node has the same token as bad one?
>
> Is it an 'documented' operation? I haven't seen it in wiki
links(Operation).
>
> ---------END----------
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Monday, January 18, 2010 11:22 PM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Re: Re: replace a bad node through bootstrapping
>
> yes
>
> On Mon, Jan 18, 2010 at 4:37 AM, XL.Pan <pa...@sina.com> wrote:
>> Hi Jonathan:
>> "the old node can be the replacement, as long as you change its IP
> address"
>>
>> Do you mean that the operations to replace a bad node is :
>> 1) choose a new machine which has the same configuration, eg.
> InitialToken, and has a different IP address;
>> 2) start the new machine, which will start boostrapping;
>> 3) After bootstrapping, the new machine will restore the data as before.
>>
>> (All nodes' InitialToken are set manully)
>>
>> I have tried in this way and that looks ok. Is this a good way? :-)
>> Thanks !!
>>
>>
>> ------------------
>> XL.Pan
>> 2010-01-18
>>
>> -------------------------------------------------------------
>> 发件人：Jonathan Ellis
>> 发送日期：2010-01-15 11:12:05
>> 收件人：cassandra-user
>> 抄送：
>> 主题：Re: Re: replace a bad node through bootstrapping
>>
>> On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
>> <ma...@gmail.com> wrote:
>>> If a node's data has been damaged, you cannot use new node replace old
> one directly, unless 'removetoken' first.
>>>
>>> But, (suppose node A is dead)
>>> 'removetoken' will complement missing replica due A's death first, it
> will generate lot data on other nodes, say it's B, C, D
>>> After add new node and copy data from other node through bootstrapping,
> you have to 'cleanup' data just
>>> generate from ' removetoken ' on B, C, D
>>>
>>> So, B/C/D will have heavy I/O load (half of them is waste) due to repair
> A, in pan's case, it will be 5TB (and will cause days...)
>>>
>>> Pan try to invent a method to repair A directly through streaming, and
> have less impact on other nodes.
>>
>> Thanks for clarifying that.
>>
>> I thought we agreed in your last thread about this that bootstrapping
>> a replacement node (the old node can be the replacement, as long as
>> you change its IP address) first, then removing the entry for the dead
>> one, would be a reasonable procedure here.
>>
>> -Jonathan
>>
>
>
Re: Re: replace a bad node through bootstrapping
Posted by Jonathan Ellis <jb...@gmail.com>.
This is described in the "Handling failure" section of the Operations page.
I believe it will work even if you use the same token as the old node, yes.
-Jonathan
2010/1/18 Michael Lee <ma...@gmail.com>:
> Even good node has the same token as bad one?
>
> Is it an 'documented' operation? I haven't seen it in wiki links(Operation).
>
> ---------END----------
>
> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Monday, January 18, 2010 11:22 PM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Re: Re: replace a bad node through bootstrapping
>
> yes
>
> On Mon, Jan 18, 2010 at 4:37 AM, XL.Pan <pa...@sina.com> wrote:
>> Hi Jonathan:
>> "the old node can be the replacement, as long as you change its IP
> address"
>>
>> Do you mean that the operations to replace a bad node is :
>> 1) choose a new machine which has the same configuration, eg.
> InitialToken, and has a different IP address;
>> 2) start the new machine, which will start boostrapping;
>> 3) After bootstrapping, the new machine will restore the data as before.
>>
>> (All nodes' InitialToken are set manully)
>>
>> I have tried in this way and that looks ok. Is this a good way? :-)
>> Thanks !!
>>
>>
>> ------------------
>> XL.Pan
>> 2010-01-18
>>
>> -------------------------------------------------------------
>> 发件人：Jonathan Ellis
>> 发送日期：2010-01-15 11:12:05
>> 收件人：cassandra-user
>> 抄送：
>> 主题：Re: Re: replace a bad node through bootstrapping
>>
>> On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
>> <ma...@gmail.com> wrote:
>>> If a node's data has been damaged, you cannot use new node replace old
> one directly, unless 'removetoken' first.
>>>
>>> But, (suppose node A is dead)
>>> 'removetoken' will complement missing replica due A's death first, it
> will generate lot data on other nodes, say it's B, C, D
>>> After add new node and copy data from other node through bootstrapping,
> you have to 'cleanup' data just
>>> generate from ' removetoken ' on B, C, D
>>>
>>> So, B/C/D will have heavy I/O load (half of them is waste) due to repair
> A, in pan's case, it will be 5TB (and will cause days...)
>>>
>>> Pan try to invent a method to repair A directly through streaming, and
> have less impact on other nodes.
>>
>> Thanks for clarifying that.
>>
>> I thought we agreed in your last thread about this that bootstrapping
>> a replacement node (the old node can be the replacement, as long as
>> you change its IP address) first, then removing the entry for the dead
>> one, would be a reasonable procedure here.
>>
>> -Jonathan
>>
>
>
Re: Re: replace a bad node through bootstrapping
Posted by Michael Lee <ma...@gmail.com>.
Even good node has the same token as bad one?
Is it an 'documented' operation? I haven't seen it in wiki links(Operation).
---------END----------
-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Monday, January 18, 2010 11:22 PM
To: cassandra-user@incubator.apache.org
Subject: Re: Re: Re: replace a bad node through bootstrapping
yes
On Mon, Jan 18, 2010 at 4:37 AM, XL.Pan <pa...@sina.com> wrote:
> Hi Jonathan:
> "the old node can be the replacement, as long as you change its IP
address"
>
> Do you mean that the operations to replace a bad node is :
> 1) choose a new machine which has the same configuration, eg.
InitialToken, and has a different IP address;
> 2) start the new machine, which will start boostrapping;
> 3) After bootstrapping, the new machine will restore the data as before.
>
> (All nodes' InitialToken are set manully)
>
> I have tried in this way and that looks ok. Is this a good way? :-)
> Thanks !!
>
>
> ------------------
> XL.Pan
> 2010-01-18
>
> -------------------------------------------------------------
> 发件人：Jonathan Ellis
> 发送日期：2010-01-15 11:12:05
> 收件人：cassandra-user
> 抄送：
> 主题：Re: Re: replace a bad node through bootstrapping
>
> On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
> <ma...@gmail.com> wrote:
>> If a node's data has been damaged, you cannot use new node replace old
one directly, unless 'removetoken' first.
>>
>> But, (suppose node A is dead)
>> 'removetoken' will complement missing replica due A's death first, it
will generate lot data on other nodes, say it's B, C, D
>> After add new node and copy data from other node through bootstrapping,
you have to 'cleanup' data just
>> generate from ' removetoken ' on B, C, D
>>
>> So, B/C/D will have heavy I/O load (half of them is waste) due to repair
A, in pan's case, it will be 5TB (and will cause days...)
>>
>> Pan try to invent a method to repair A directly through streaming, and
have less impact on other nodes.
>
> Thanks for clarifying that.
>
> I thought we agreed in your last thread about this that bootstrapping
> a replacement node (the old node can be the replacement, as long as
> you change its IP address) first, then removing the entry for the dead
> one, would be a reasonable procedure here.
>
> -Jonathan
>
Re: Re: Re: replace a bad node through bootstrapping
Posted by Jonathan Ellis <jb...@gmail.com>.
yes
On Mon, Jan 18, 2010 at 4:37 AM, XL.Pan <pa...@sina.com> wrote:
> Hi Jonathan:
> "the old node can be the replacement, as long as you change its IP address"
>
> Do you mean that the operations to replace a bad node is :
> 1) choose a new machine which has the same configuration, eg. InitialToken, and has a different IP address;
> 2) start the new machine, which will start boostrapping;
> 3) After bootstrapping, the new machine will restore the data as before.
>
> (All nodes' InitialToken are set manully)
>
> I have tried in this way and that looks ok. Is this a good way? :-)
> Thanks !!
>
>
> ------------------
> XL.Pan
> 2010-01-18
>
> -------------------------------------------------------------
> 发件人：Jonathan Ellis
> 发送日期：2010-01-15 11:12:05
> 收件人：cassandra-user
> 抄送：
> 主题：Re: Re: replace a bad node through bootstrapping
>
> On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
> <ma...@gmail.com> wrote:
>> If a node's data has been damaged, you cannot use new node replace old one directly, unless 'removetoken' first.
>>
>> But, (suppose node A is dead)
>> 'removetoken' will complement missing replica due A's death first, it will generate lot data on other nodes, say it's B, C, D
>> After add new node and copy data from other node through bootstrapping, you have to 'cleanup' data just
>> generate from ' removetoken ' on B, C, D
>>
>> So, B/C/D will have heavy I/O load (half of them is waste) due to repair A, in pan's case, it will be 5TB (and will cause days...)
>>
>> Pan try to invent a method to repair A directly through streaming, and have less impact on other nodes.
>
> Thanks for clarifying that.
>
> I thought we agreed in your last thread about this that bootstrapping
> a replacement node (the old node can be the replacement, as long as
> you change its IP address) first, then removing the entry for the dead
> one, would be a reasonable procedure here.
>
> -Jonathan
>
Re: Re: Re: replace a bad node through bootstrapping
Posted by "XL.Pan" <pa...@sina.com>.
Hi Jonathan:
"the old node can be the replacement, as long as you change its IP address"
Do you mean that the operations to replace a bad node is :
1) choose a new machine which has the same configuration, eg. InitialToken, and has a different IP address;
2) start the new machine, which will start boostrapping;
3) After bootstrapping, the new machine will restore the data as before.
(All nodes' InitialToken are set manully)
I have tried in this way and that looks ok. Is this a good way? :-)
Thanks !!
------------------
XL.Pan
2010-01-18
-------------------------------------------------------------
发件人：Jonathan Ellis
发送日期：2010-01-15 11:12:05
收件人：cassandra-user
抄送：
主题：Re: Re: replace a bad node through bootstrapping
On Thu, Jan 14, 2010 at 9:02 PM, Michael Lee
<ma...@gmail.com> wrote:
> If a node's data has been damaged, you cannot use new node replace old one directly, unless 'removetoken' first.
>
> But, (suppose node A is dead)
> 'removetoken' will complement missing replica due A's death first, it will generate lot data on other nodes, say it's B, C, D
> After add new node and copy data from other node through bootstrapping, you have to 'cleanup' data just
> generate from ' removetoken ' on B, C, D
>
> So, B/C/D will have heavy I/O load (half of them is waste) due to repair A, in pan's case, it will be 5TB (and will cause days...)
>
> Pan try to invent a method to repair A directly through streaming, and have less impact on other nodes.
Thanks for clarifying that.
I thought we agreed in your last thread about this that bootstrapping
a replacement node (the old node can be the replacement, as long as
you change its IP address) first, then removing the entry for the dead
one, would be a reasonable procedure here.
-Jonathan
RE: Re: replace a bad node through bootstrapping
Posted by Michael Lee <ma...@gmail.com>.
I' am Pan's collogue, allow me make it clear...
Pan's problem is:
If a node's data has been damaged, you cannot use new node replace old one directly, unless 'removetoken' first.
But, (suppose node A is dead)
'removetoken' will complement missing replica due A's death first, it will generate lot data on other nodes, say it's B, C, D
After add new node and copy data from other node through bootstrapping, you have to 'cleanup' data just
generate from ' removetoken ' on B, C, D
So, B/C/D will have heavy I/O load (half of them is waste) due to repair A, in pan's case, it will be 5TB (and will cause days...)
Pan try to invent a method to repair A directly through streaming, and have less impact on other nodes.
---------END----------
-----Original Message-----
From: XL.Pan [mailto:pan_xiaolei@sina.com]
Sent: Friday, January 15, 2010 10:23 AM
To: cassandra-user; cassandra-user
Subject: Re: Re: replace a bad node through bootstrapping
--------------------------------------------------
| Range changes |
| Bootstrap |
| Adding new nodes is called "bootstrapping." |
-------------------------------------------------
Do you mean that "bootstrapping" is designed for adding new nodes only?
I think the bootstrapping idea is good enough to do something else, for example that restoring the data in a bad node, though it needs some modification if that.
What's the difference between a new one which is NOT in the ring before and a new one which is in the ring before?
I think there are some similarities and differences. (let new one called N, and the replaced one called R)
* similarities:
Both N and R have no available data, as a result that both of which need to copy data from the replication soureces.
* differences:
1) About writing while startup
N is not seen before and of couse it has no handoff data in other nodes. As a result that it should serve for writing which is routed from other nodes while coping data.
R is seen before and it has handoff data in other nodes, so it will not care about losing data while coping data from other sources and it will receive the handoff data after startup. That means R has no need to serve for writing at that time.
2) About the selection of replication sources
A-->B
| |
D<--C
N want's to insert between B and C, so N knows that it can get data from C, D, A. After bootstrapping, the ring will be:
A-->B
| |
| N
| |
D<--C
Then the node N is down and replaced with R. Because the R has seen a different ring, it will select B and C.
>From the comparison, I think it's possible that replacing a bad node and restore the data through bootstrapping.
------------------
XL.Pan
2010-01-15
-------------------------------------------------------------
发件人：Jonathan Ellis
发送日期：2010-01-15 00:51:57
收件人：cassandra-user
抄送：
主题：Re: replace a bad node through bootstrapping
On Thu, Jan 14, 2010 at 6:30 AM, XL.Pan <pa...@sina.com> wrote:
> *Why not the standard boostrap?
http://wiki.apache.org/cassandra/Operations says that boostrap is the
preferred method for handling node replacement. Please read how that
describes how to handle things because your description of how
bootstrap works is very off base.
-Joanthan
Re: Re: replace a bad node through bootstrapping
Posted by "XL.Pan" <pa...@sina.com>.
--------------------------------------------------
| Range changes |
| Bootstrap |
| Adding new nodes is called "bootstrapping." |
-------------------------------------------------
Do you mean that "bootstrapping" is designed for adding new nodes only?
I think the bootstrapping idea is good enough to do something else, for example that restoring the data in a bad node, though it needs some modification if that.
What's the difference between a new one which is NOT in the ring before and a new one which is in the ring before?
I think there are some similarities and differences. (let new one called N, and the replaced one called R)
* similarities:
Both N and R have no available data, as a result that both of which need to copy data from the replication soureces.
* differences:
1) About writing while startup
N is not seen before and of couse it has no handoff data in other nodes. As a result that it should serve for writing which is routed from other nodes while coping data.
R is seen before and it has handoff data in other nodes, so it will not care about losing data while coping data from other sources and it will receive the handoff data after startup. That means R has no need to serve for writing at that time.
2) About the selection of replication sources
A-->B
| |
D<--C
N want's to insert between B and C, so N knows that it can get data from C, D, A. After bootstrapping, the ring will be:
A-->B
| |
| N
| |
D<--C
Then the node N is down and replaced with R. Because the R has seen a different ring, it will select B and C.
From the comparison, I think it's possible that replacing a bad node and restore the data through bootstrapping.
------------------
XL.Pan
2010-01-15
-------------------------------------------------------------
发件人：Jonathan Ellis
发送日期：2010-01-15 00:51:57
收件人：cassandra-user
抄送：
主题：Re: replace a bad node through bootstrapping
On Thu, Jan 14, 2010 at 6:30 AM, XL.Pan <pa...@sina.com> wrote:
> *Why not the standard boostrap?
http://wiki.apache.org/cassandra/Operations says that boostrap is the
preferred method for handling node replacement. Please read how that
describes how to handle things because your description of how
bootstrap works is very off base.
-Joanthan
Re: replace a bad node through bootstrapping
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Jan 14, 2010 at 6:30 AM, XL.Pan <pa...@sina.com> wrote:
> *Why not the standard boostrap?
http://wiki.apache.org/cassandra/Operations says that boostrap is the
preferred method for handling node replacement. Please read how that
describes how to handle things because your description of how
bootstrap works is very off base.
-Joanthan