You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Tomas Vondra <tv...@fuzzy.cz> on 2011/04/24 01:46:04 UTC
decommissioning a wrong node
Hi all,
I've set up a ring with three nodes to play with. I've tried to
decommision a node, but it seems a different node is decommisioned.
Example - the nodes are running:
~/cassandra $ nodetool -h 10.0.0.1 ring
Address Status State Load Owns Token
729261389019385599444...
10.0.0.3 Up Normal 62,28 KB 72,05% 253757231779485488680...
10.0.0.1 Up Normal 53,67 KB 25,16% 681842499981089716433...
10.0.0.2 Up Normal 57,92 KB 2,79% 729261389019385599444...
And then I would like to decommission node with IP 10.0.0.2, so I do this
~/cassandra $ nodetool -h 10.0.0.2 decommission
which for some reason does this:
~/cassandra $ nodetool -h 10.0.0.1 ring
Address Status State Load Owns Token
7292613890193855994...
10.0.0.3 Up Normal 62,28 KB 72,05% 2537572317794854886...
10.0.0.1 Up Leaving 53,67 KB 25,16% 6818424999810897164...
10.0.0.2 Up Normal 57,92 KB 2,79% 7292613890193855994...
and in the end
~/cassandra $ nodetool -h 10.0.0.1 ring
Address Status State Load Owns Token
7292613890193855994...
10.0.0.3 Up Normal 62,28 KB 72,05% 2537572317794854886...
10.0.0.2 Up Normal 57,92 KB 27,95% 7292613890193855994...
So a different node (with IP 10.0.0.1, not 10.0.0.2) is decommissioned.
BTW all the nodes are running on the same machine and are set up using
virtual IPs. Not sure if this is important (I don't think it should).
regards
Tomas
Re: decommissioning a wrong node
Posted by aaron morton <aa...@thelastpickle.com>.
> BTW all the nodes are running on the same machine and are set up using
> virtual IPs. Not sure if this is important (I don't think it shoul
Yup :)
JMX binds to 127.0.0.1, but different ports as specified in cassandra-env.sh . Include the port number on the nodetool command line, otherwise you are just talking to the instance on the default JMX port.
Aaron
On 24 Apr 2011, at 11:46, Tomas Vondra wrote:
> Hi all,
>
> I've set up a ring with three nodes to play with. I've tried to
> decommision a node, but it seems a different node is decommisioned.
>
> Example - the nodes are running:
>
> ~/cassandra $ nodetool -h 10.0.0.1 ring
> Address Status State Load Owns Token
> 729261389019385599444...
> 10.0.0.3 Up Normal 62,28 KB 72,05% 253757231779485488680...
> 10.0.0.1 Up Normal 53,67 KB 25,16% 681842499981089716433...
> 10.0.0.2 Up Normal 57,92 KB 2,79% 729261389019385599444...
>
> And then I would like to decommission node with IP 10.0.0.2, so I do this
>
> ~/cassandra $ nodetool -h 10.0.0.2 decommission
>
> which for some reason does this:
>
> ~/cassandra $ nodetool -h 10.0.0.1 ring
> Address Status State Load Owns Token
> 7292613890193855994...
> 10.0.0.3 Up Normal 62,28 KB 72,05% 2537572317794854886...
> 10.0.0.1 Up Leaving 53,67 KB 25,16% 6818424999810897164...
> 10.0.0.2 Up Normal 57,92 KB 2,79% 7292613890193855994...
>
> and in the end
>
> ~/cassandra $ nodetool -h 10.0.0.1 ring
> Address Status State Load Owns Token
> 7292613890193855994...
> 10.0.0.3 Up Normal 62,28 KB 72,05% 2537572317794854886...
> 10.0.0.2 Up Normal 57,92 KB 27,95% 7292613890193855994...
>
> So a different node (with IP 10.0.0.1, not 10.0.0.2) is decommissioned.
>
>
> BTW all the nodes are running on the same machine and are set up using
> virtual IPs. Not sure if this is important (I don't think it should).
>
> regards
> Tomas
Re: decommissioning a wrong node
Posted by Tomas Vondra <tv...@fuzzy.cz>.
Dne 26.4.2011 09:21, aaron morton napsal(a):
> There is the fabled
>
> java.rmi.server.hostname
>
> http://blog.reactive.org/2011/02/connecting-to-cassandra-jmx-via-ssh.html
> http://download.oracle.com/javase/1.4.2/docs/guide/rmi/javarmiproperties.html
>
> Not sure if anyone has got it working correctly.
Thanks, at least I know I'm not the only one who noticed this issue.
Tomas
Re: decommissioning a wrong node
Posted by aaron morton <aa...@thelastpickle.com>.
There is the fabled
java.rmi.server.hostname
http://blog.reactive.org/2011/02/connecting-to-cassandra-jmx-via-ssh.html
http://download.oracle.com/javase/1.4.2/docs/guide/rmi/javarmiproperties.html
Not sure if anyone has got it working correctly.
Aaron
On 25 Apr 2011, at 03:51, Edward Capriolo wrote:
> On Sun, Apr 24, 2011 at 6:07 AM, Peter Schuller
> <pe...@infidyne.com> wrote:
>>> Is there a way to bind the JMX to a specified IP only? It seems there's
>>> just 'com.sun.management.jmxremote.port' and no way to specify a host.
>>
>> I don't think so, or at least past googling indicated several people
>> wanting to do this but not finding answers. It's extremely annoying;
>> e.g. the common case of wanting to listen on 127.0.0.1 but no public
>> interfaces...
>>
>> --
>> / Peter Schuller
>>
>
> It can not be done. There is a sun bug id on this somewhere. The good
> news is that even if JMX binds to an IP you do not want it to, the
> redirect by hostname and dynamic port choosing in the JMX protocol
> will probably cause it not to work "security my complexity".
Re: decommissioning a wrong node
Posted by Edward Capriolo <ed...@gmail.com>.
On Sun, Apr 24, 2011 at 6:07 AM, Peter Schuller
<pe...@infidyne.com> wrote:
>> Is there a way to bind the JMX to a specified IP only? It seems there's
>> just 'com.sun.management.jmxremote.port' and no way to specify a host.
>
> I don't think so, or at least past googling indicated several people
> wanting to do this but not finding answers. It's extremely annoying;
> e.g. the common case of wanting to listen on 127.0.0.1 but no public
> interfaces...
>
> --
> / Peter Schuller
>
It can not be done. There is a sun bug id on this somewhere. The good
news is that even if JMX binds to an IP you do not want it to, the
redirect by hostname and dynamic port choosing in the JMX protocol
will probably cause it not to work "security my complexity".
Re: decommissioning a wrong node
Posted by Peter Schuller <pe...@infidyne.com>.
> Is there a way to bind the JMX to a specified IP only? It seems there's
> just 'com.sun.management.jmxremote.port' and no way to specify a host.
I don't think so, or at least past googling indicated several people
wanting to do this but not finding answers. It's extremely annoying;
e.g. the common case of wanting to listen on 127.0.0.1 but no public
interfaces...
--
/ Peter Schuller
Re: decommissioning a wrong node
Posted by Tomas Vondra <tv...@fuzzy.cz>.
Dne 24.4.2011 04:32, Edward Capriolo napsal(a):
> 2011/4/23 Tomas Vondra <tv...@fuzzy.cz>:
>>
>> BTW all the nodes are running on the same machine and are set up using
>> virtual IPs. Not sure if this is important (I don't think it should).
>>
>> regards
>> Tomas
>>
>
> Ah ha. JMX binds to it's port on EVERY interface on the system. So
> since you are not specifying a port it is defaulting to 8080 and
> decommissioning that one. You probably set the other ones up as 8001
> 8002 but I am no psychic.
Oh, the nodetool uses JMX and not the storage port directly? Hm, that
probably makes sense.
And yes, you're right - the nodes use different ports (8080, 8082 and
8083). So nodetool connects to 10.0.0.2 and uses the default JMX port
(8080, corresponding to node1) and sends 'decommission' command. Once I
specify the port the right node is decommissioned ...
Is there a way to bind the JMX to a specified IP only? It seems there's
just 'com.sun.management.jmxremote.port' and no way to specify a host.
regards
Tomas
Re: decommissioning a wrong node
Posted by Edward Capriolo <ed...@gmail.com>.
2011/4/23 Tomas Vondra <tv...@fuzzy.cz>:
> Hi all,
>
> I've set up a ring with three nodes to play with. I've tried to
> decommision a node, but it seems a different node is decommisioned.
>
> Example - the nodes are running:
>
> ~/cassandra $ nodetool -h 10.0.0.1 ring
> Address Status State Load Owns Token
> 729261389019385599444...
> 10.0.0.3 Up Normal 62,28 KB 72,05% 253757231779485488680...
> 10.0.0.1 Up Normal 53,67 KB 25,16% 681842499981089716433...
> 10.0.0.2 Up Normal 57,92 KB 2,79% 729261389019385599444...
>
> And then I would like to decommission node with IP 10.0.0.2, so I do this
>
> ~/cassandra $ nodetool -h 10.0.0.2 decommission
>
> which for some reason does this:
>
> ~/cassandra $ nodetool -h 10.0.0.1 ring
> Address Status State Load Owns Token
> 7292613890193855994...
> 10.0.0.3 Up Normal 62,28 KB 72,05% 2537572317794854886...
> 10.0.0.1 Up Leaving 53,67 KB 25,16% 6818424999810897164...
> 10.0.0.2 Up Normal 57,92 KB 2,79% 7292613890193855994...
>
> and in the end
>
> ~/cassandra $ nodetool -h 10.0.0.1 ring
> Address Status State Load Owns Token
> 7292613890193855994...
> 10.0.0.3 Up Normal 62,28 KB 72,05% 2537572317794854886...
> 10.0.0.2 Up Normal 57,92 KB 27,95% 7292613890193855994...
>
> So a different node (with IP 10.0.0.1, not 10.0.0.2) is decommissioned.
>
>
> BTW all the nodes are running on the same machine and are set up using
> virtual IPs. Not sure if this is important (I don't think it should).
>
> regards
> Tomas
>
Ah ha. JMX binds to it's port on EVERY interface on the system. So
since you are not specifying a port it is defaulting to 8080 and
decommissioning that one. You probably set the other ones up as 8001
8002 but I am no psychic.