You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Robin Verlangen <ro...@us2.nl> on 2012/11/09 14:36:42 UTC
Remove crashed node
Hi there,
We have had a crashed node that is currently removed from the rack. However
when I try a schema upgrade / truncate operation it complains of the
unreachable node. I tried the removetoken, but that didn't resolve.
Any ideas on how to fix this?
Best regards,
Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
<http://goo.gl/Lt7BC>
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
Re: Remove crashed node
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
I am glad to here that.
You're very welcome.
Alain
2012/11/19 Robin Verlangen <ro...@us2.nl>
> Quite some time passed, but Alain, it worked. Thank you very much.
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> <http://goo.gl/Lt7BC>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>
>
> On Fri, Nov 9, 2012 at 4:21 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>
>> You have to install mx4j-tools.jar.
>>
>> http://wiki.apache.org/cassandra/Operations#Monitoring_with_MX4J
>>
>> It's a java tool, so it is usable on both windows and linux.
>>
>> Here is the link to dl mx4j-tool.jar :
>> http://www.java2s.com/Code/JarDownload/mx4j/mx4j-tools-3.0.2.jar.zip
>>
>> unzip it and add it to the path of your cassandra libraries, restart the
>> node where it is installed and you should be ok.
>>
>> Alain
>>
>>
>> 2012/11/9 Robin Verlangen <ro...@us2.nl>
>>
>>> Hi Alain,
>>>
>>> How can I access that? Web browser does not seem to work. Do I need any
>>> software to login? If so, what is proper software for Windows?
>>>
>>> Best regards,
>>>
>>> Robin Verlangen
>>> *Software engineer*
>>> *
>>> *
>>> W http://www.robinverlangen.nl
>>> E robin@us2.nl
>>>
>>> <http://goo.gl/Lt7BC>
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>>
>>>
>>> On Fri, Nov 9, 2012 at 2:50 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>>>
>>>> On cassandra-cli if you "describe cluster;" I guess you will see an
>>>> "UNREACHABLE" node.
>>>>
>>>> If you do so, there is a way to remove this unreachable node.
>>>>
>>>> Go to the JMX management console (ip_of_one_up_node:8081 by default)
>>>>
>>>> Then go to the org.apache.cassandra.net:type=Gossiper link and use the
>>>> "unsafeAssassinateEndpoint" input. Fill it with the ip of the down
>>>> node and invoke the function.
>>>>
>>>> "nodetool gossipinfo" should now tell you that this node has left the
>>>> ring and let you truncate or whatever you need to do.
>>>>
>>>> Use this carefully this function is composed of the "unsafe" and
>>>> "assassinate" words because it forces the node to go out of the ring
>>>> without any check or replication.
>>>>
>>>> Alain
>>>>
>>>>
>>>> 2012/11/9 Robin Verlangen <ro...@us2.nl>
>>>>
>>>>> Hi there,
>>>>>
>>>>> We have had a crashed node that is currently removed from the rack.
>>>>> However when I try a schema upgrade / truncate operation it complains of
>>>>> the unreachable node. I tried the removetoken, but that didn't resolve.
>>>>>
>>>>> Any ideas on how to fix this?
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Robin Verlangen
>>>>> *Software engineer*
>>>>> *
>>>>> *
>>>>> W http://www.robinverlangen.nl
>>>>> E robin@us2.nl
>>>>>
>>>>> <http://goo.gl/Lt7BC>
>>>>>
>>>>> Disclaimer: The information contained in this message and attachments
>>>>> is intended solely for the attention and use of the named addressee and may
>>>>> be confidential. If you are not the intended recipient, you are reminded
>>>>> that the information remains the property of the sender. You must not use,
>>>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>>>> received this message in error, please contact the sender immediately and
>>>>> irrevocably delete this message and any copies.
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Remove crashed node
Posted by Robin Verlangen <ro...@us2.nl>.
Quite some time passed, but Alain, it worked. Thank you very much.
Best regards,
Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
<http://goo.gl/Lt7BC>
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
On Fri, Nov 9, 2012 at 4:21 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> You have to install mx4j-tools.jar.
>
> http://wiki.apache.org/cassandra/Operations#Monitoring_with_MX4J
>
> It's a java tool, so it is usable on both windows and linux.
>
> Here is the link to dl mx4j-tool.jar :
> http://www.java2s.com/Code/JarDownload/mx4j/mx4j-tools-3.0.2.jar.zip
>
> unzip it and add it to the path of your cassandra libraries, restart the
> node where it is installed and you should be ok.
>
> Alain
>
>
> 2012/11/9 Robin Verlangen <ro...@us2.nl>
>
>> Hi Alain,
>>
>> How can I access that? Web browser does not seem to work. Do I need any
>> software to login? If so, what is proper software for Windows?
>>
>> Best regards,
>>
>> Robin Verlangen
>> *Software engineer*
>> *
>> *
>> W http://www.robinverlangen.nl
>> E robin@us2.nl
>>
>> <http://goo.gl/Lt7BC>
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>>
>>
>> On Fri, Nov 9, 2012 at 2:50 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>>
>>> On cassandra-cli if you "describe cluster;" I guess you will see an
>>> "UNREACHABLE" node.
>>>
>>> If you do so, there is a way to remove this unreachable node.
>>>
>>> Go to the JMX management console (ip_of_one_up_node:8081 by default)
>>>
>>> Then go to the org.apache.cassandra.net:type=Gossiper link and use the "unsafeAssassinateEndpoint"
>>> input. Fill it with the ip of the down node and invoke the function.
>>>
>>> "nodetool gossipinfo" should now tell you that this node has left the
>>> ring and let you truncate or whatever you need to do.
>>>
>>> Use this carefully this function is composed of the "unsafe" and
>>> "assassinate" words because it forces the node to go out of the ring
>>> without any check or replication.
>>>
>>> Alain
>>>
>>>
>>> 2012/11/9 Robin Verlangen <ro...@us2.nl>
>>>
>>>> Hi there,
>>>>
>>>> We have had a crashed node that is currently removed from the rack.
>>>> However when I try a schema upgrade / truncate operation it complains of
>>>> the unreachable node. I tried the removetoken, but that didn't resolve.
>>>>
>>>> Any ideas on how to fix this?
>>>>
>>>> Best regards,
>>>>
>>>> Robin Verlangen
>>>> *Software engineer*
>>>> *
>>>> *
>>>> W http://www.robinverlangen.nl
>>>> E robin@us2.nl
>>>>
>>>> <http://goo.gl/Lt7BC>
>>>>
>>>> Disclaimer: The information contained in this message and attachments
>>>> is intended solely for the attention and use of the named addressee and may
>>>> be confidential. If you are not the intended recipient, you are reminded
>>>> that the information remains the property of the sender. You must not use,
>>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>>> received this message in error, please contact the sender immediately and
>>>> irrevocably delete this message and any copies.
>>>>
>>>>
>>>
>>
>
Re: Remove crashed node
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
You have to install mx4j-tools.jar.
http://wiki.apache.org/cassandra/Operations#Monitoring_with_MX4J
It's a java tool, so it is usable on both windows and linux.
Here is the link to dl mx4j-tool.jar :
http://www.java2s.com/Code/JarDownload/mx4j/mx4j-tools-3.0.2.jar.zip
unzip it and add it to the path of your cassandra libraries, restart the
node where it is installed and you should be ok.
Alain
2012/11/9 Robin Verlangen <ro...@us2.nl>
> Hi Alain,
>
> How can I access that? Web browser does not seem to work. Do I need any
> software to login? If so, what is proper software for Windows?
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> <http://goo.gl/Lt7BC>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>
>
> On Fri, Nov 9, 2012 at 2:50 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>
>> On cassandra-cli if you "describe cluster;" I guess you will see an
>> "UNREACHABLE" node.
>>
>> If you do so, there is a way to remove this unreachable node.
>>
>> Go to the JMX management console (ip_of_one_up_node:8081 by default)
>>
>> Then go to the org.apache.cassandra.net:type=Gossiper link and use the "unsafeAssassinateEndpoint"
>> input. Fill it with the ip of the down node and invoke the function.
>>
>> "nodetool gossipinfo" should now tell you that this node has left the
>> ring and let you truncate or whatever you need to do.
>>
>> Use this carefully this function is composed of the "unsafe" and
>> "assassinate" words because it forces the node to go out of the ring
>> without any check or replication.
>>
>> Alain
>>
>>
>> 2012/11/9 Robin Verlangen <ro...@us2.nl>
>>
>>> Hi there,
>>>
>>> We have had a crashed node that is currently removed from the rack.
>>> However when I try a schema upgrade / truncate operation it complains of
>>> the unreachable node. I tried the removetoken, but that didn't resolve.
>>>
>>> Any ideas on how to fix this?
>>>
>>> Best regards,
>>>
>>> Robin Verlangen
>>> *Software engineer*
>>> *
>>> *
>>> W http://www.robinverlangen.nl
>>> E robin@us2.nl
>>>
>>> <http://goo.gl/Lt7BC>
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>>
>>
>
Re: Remove crashed node
Posted by Robin Verlangen <ro...@us2.nl>.
Hi Alain,
How can I access that? Web browser does not seem to work. Do I need any
software to login? If so, what is proper software for Windows?
Best regards,
Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
<http://goo.gl/Lt7BC>
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
On Fri, Nov 9, 2012 at 2:50 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> On cassandra-cli if you "describe cluster;" I guess you will see an
> "UNREACHABLE" node.
>
> If you do so, there is a way to remove this unreachable node.
>
> Go to the JMX management console (ip_of_one_up_node:8081 by default)
>
> Then go to the org.apache.cassandra.net:type=Gossiper link and use the "unsafeAssassinateEndpoint"
> input. Fill it with the ip of the down node and invoke the function.
>
> "nodetool gossipinfo" should now tell you that this node has left the ring
> and let you truncate or whatever you need to do.
>
> Use this carefully this function is composed of the "unsafe" and
> "assassinate" words because it forces the node to go out of the ring
> without any check or replication.
>
> Alain
>
>
> 2012/11/9 Robin Verlangen <ro...@us2.nl>
>
>> Hi there,
>>
>> We have had a crashed node that is currently removed from the rack.
>> However when I try a schema upgrade / truncate operation it complains of
>> the unreachable node. I tried the removetoken, but that didn't resolve.
>>
>> Any ideas on how to fix this?
>>
>> Best regards,
>>
>> Robin Verlangen
>> *Software engineer*
>> *
>> *
>> W http://www.robinverlangen.nl
>> E robin@us2.nl
>>
>> <http://goo.gl/Lt7BC>
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>>
>
Re: Remove crashed node
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
On cassandra-cli if you "describe cluster;" I guess you will see an
"UNREACHABLE" node.
If you do so, there is a way to remove this unreachable node.
Go to the JMX management console (ip_of_one_up_node:8081 by default)
Then go to the org.apache.cassandra.net:type=Gossiper link and use the
"unsafeAssassinateEndpoint"
input. Fill it with the ip of the down node and invoke the function.
"nodetool gossipinfo" should now tell you that this node has left the ring
and let you truncate or whatever you need to do.
Use this carefully this function is composed of the "unsafe" and
"assassinate" words because it forces the node to go out of the ring
without any check or replication.
Alain
2012/11/9 Robin Verlangen <ro...@us2.nl>
> Hi there,
>
> We have had a crashed node that is currently removed from the rack.
> However when I try a schema upgrade / truncate operation it complains of
> the unreachable node. I tried the removetoken, but that didn't resolve.
>
> Any ideas on how to fix this?
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> <http://goo.gl/Lt7BC>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>