You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Tim Dunphy <bl...@gmail.com> on 2014/10/26 22:56:29 UTC

decommissioning a cassandra node

Hey all,

 I'm trying to decommission a node.

 First I'm getting a status:

[root@beta-new:/usr/local] #nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns    Host ID
          Rack
UN  162.243.86.41   1.08 MB    1       0.1%
 e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
UL  162.243.109.94  1.28 MB    256     99.9%
fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1


But when I try to decommission the node I get this message:

[root@beta-new:/usr/local] #nodetool -h 162.243.86.41 decommission
nodetool: Failed to connect to '162.243.86.41:7199' -
NoSuchObjectException: 'no such object in table'.

Yet I can telnet to that host on that port just fine:

[root@beta-new:/usr/local] #telnet 162.243.86.41 7199
Trying 162.243.86.41...
Connected to 162.243.86.41.
Escape character is '^]'.


And I have verified that cassandra is running and accessible via cqlsh on
the other machine.

What could be going wrong?

Thanks
Tim


-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: decommissioning a cassandra node

Posted by Tim Dunphy <bl...@gmail.com>.
>
> The node with IP 94 is leaving. Maybe something wrong happens during
> streaming data. You could use "nodetool netstats" on both nodes to monitor
> if there is any streaming connection stuck.
> Indeed, you could force remove the leaving node by shutting down it
> directly. Then, perform "nodetool removenode" to remove dead node. But you
> should understand you're taking the risk to lose data if your RF in cluster
> is lower than 3 and data have not been fully synced. Therefore, remember to
> sync data using repair before you're going to remove/decommission the node
> in cluster.



Hi Colin,

 Ok that's good advice. Thanks for your help! I'll give that a shot and see
what I can do.

Thanks
Tim

On Mon, Oct 27, 2014 at 11:17 AM, Colin Kuo <co...@gmail.com> wrote:

> Hi Tim,
>
> The node with IP 94 is leaving. Maybe something wrong happens during
> streaming data. You could use "nodetool netstats" on both nodes to monitor
> if there is any streaming connection stuck.
>
> Indeed, you could force remove the leaving node by shutting down it
> directly. Then, perform "nodetool removenode" to remove dead node. But you
> should understand you're taking the risk to lose data if your RF in cluster
> is lower than 3 and data have not been fully synced. Therefore, remember to
> sync data using repair before you're going to remove/decommission the node
> in cluster.
>
> Thanks!
>
> On Mon, Oct 27, 2014 at 9:55 PM, Tim Dunphy <bl...@gmail.com> wrote:
>
>> "Also, is there any document that explains what all the nodetool
>>> abbreviations (UN, UL) stand for?"
>>> --> The documentation is in the command output itself
>>> Datacenter: datacenter1
>>> =======================
>>>
>>> *Status=Up/Down*
>>> *|/ State=Normal/Leaving/Joining/Moving*--  Address         Load
>>> Tokens  Owns    Host ID                               Rack
>>> UN  162.243.86.41   1.08 MB    1       0.1%
>>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>>> UL  162.243.109.94  1.28 MB    256     99.9%
>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>>> U = Up, D = Down
>>> N = Normal, L = Leaving, J = Joining and M = Moving
>>
>>
>> Ok, got it, thanks!
>>
>> Can someone suggest a good way to fix a node that is in an UL state?
>>
>> Thanks
>> Tim
>>
>> On Mon, Oct 27, 2014 at 9:46 AM, DuyHai Doan <do...@gmail.com>
>> wrote:
>>
>>> "Also, is there any document that explains what all the nodetool
>>> abbreviations (UN, UL) stand for?"
>>>
>>> --> The documentation is in the command output itself
>>>
>>> Datacenter: datacenter1
>>> =======================
>>> *Status=Up/Down*
>>> *|/ State=Normal/Leaving/Joining/Moving*
>>> --  Address         Load       Tokens  Owns    Host ID
>>>             Rack
>>> UN  162.243.86.41   1.08 MB    1       0.1%
>>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>>> UL  162.243.109.94  1.28 MB    256     99.9%
>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>>>
>>> U = Up, D = Down
>>> N = Normal, L = Leaving, J = Joining and M = Moving
>>>
>>> On Mon, Oct 27, 2014 at 2:42 PM, Tim Dunphy <bl...@gmail.com>
>>> wrote:
>>>
>>>> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>>>>> causing the problem
>>>>
>>>>
>>>> OK, that's an interesting observation.How do you fix a node that is an
>>>> UL state? What causes this?
>>>>
>>>> Also, is there any document that explains what all the nodetool
>>>> abbreviations (UN, UL) stand for?
>>>>
>>>> On Mon, Oct 27, 2014 at 5:46 AM, jivko donev <ji...@yahoo.com>
>>>> wrote:
>>>>
>>>>> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>>>>> causing the problem.
>>>>>
>>>>>
>>>>>   On Sunday, October 26, 2014 11:57 PM, Tim Dunphy <
>>>>> bluethundr@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Hey all,
>>>>>
>>>>>  I'm trying to decommission a node.
>>>>>
>>>>>  First I'm getting a status:
>>>>>
>>>>> [root@beta-new:/usr/local] #nodetool status
>>>>> Note: Ownership information does not include topology; for complete
>>>>> information, specify a keyspace
>>>>> Datacenter: datacenter1
>>>>> =======================
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> --  Address         Load       Tokens  Owns    Host ID
>>>>>               Rack
>>>>> UN  162.243.86.41   1.08 MB    1       0.1%
>>>>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>>>>> UL  162.243.109.94  1.28 MB    256     99.9%
>>>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>>>>>
>>>>>
>>>>> But when I try to decommission the node I get this message:
>>>>>
>>>>> [root@beta-new:/usr/local] #nodetool -h 162.243.86.41 decommission
>>>>> nodetool: Failed to connect to '162.243.86.41:7199' -
>>>>> NoSuchObjectException: 'no such object in table'.
>>>>>
>>>>> Yet I can telnet to that host on that port just fine:
>>>>>
>>>>> [root@beta-new:/usr/local] #telnet 162.243.86.41 7199
>>>>> Trying 162.243.86.41...
>>>>> Connected to 162.243.86.41.
>>>>> Escape character is '^]'.
>>>>>
>>>>>
>>>>> And I have verified that cassandra is running and accessible via cqlsh
>>>>> on the other machine.
>>>>>
>>>>> What could be going wrong?
>>>>>
>>>>> Thanks
>>>>> Tim
>>>>>
>>>>>
>>>>> --
>>>>> GPG me!!
>>>>>
>>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> GPG me!!
>>>>
>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>>
>>>>
>>>
>>
>>
>> --
>> GPG me!!
>>
>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>
>>
>


-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: decommissioning a cassandra node

Posted by Colin Kuo <co...@gmail.com>.
Hi Tim,

The node with IP 94 is leaving. Maybe something wrong happens during
streaming data. You could use "nodetool netstats" on both nodes to monitor
if there is any streaming connection stuck.

Indeed, you could force remove the leaving node by shutting down it
directly. Then, perform "nodetool removenode" to remove dead node. But you
should understand you're taking the risk to lose data if your RF in cluster
is lower than 3 and data have not been fully synced. Therefore, remember to
sync data using repair before you're going to remove/decommission the node
in cluster.

Thanks!

On Mon, Oct 27, 2014 at 9:55 PM, Tim Dunphy <bl...@gmail.com> wrote:

> "Also, is there any document that explains what all the nodetool
>> abbreviations (UN, UL) stand for?"
>> --> The documentation is in the command output itself
>> Datacenter: datacenter1
>> =======================
>>
>> *Status=Up/Down*
>> *|/ State=Normal/Leaving/Joining/Moving*--  Address         Load
>> Tokens  Owns    Host ID                               Rack
>> UN  162.243.86.41   1.08 MB    1       0.1%
>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>> UL  162.243.109.94  1.28 MB    256     99.9%
>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>> U = Up, D = Down
>> N = Normal, L = Leaving, J = Joining and M = Moving
>
>
> Ok, got it, thanks!
>
> Can someone suggest a good way to fix a node that is in an UL state?
>
> Thanks
> Tim
>
> On Mon, Oct 27, 2014 at 9:46 AM, DuyHai Doan <do...@gmail.com> wrote:
>
>> "Also, is there any document that explains what all the nodetool
>> abbreviations (UN, UL) stand for?"
>>
>> --> The documentation is in the command output itself
>>
>> Datacenter: datacenter1
>> =======================
>> *Status=Up/Down*
>> *|/ State=Normal/Leaving/Joining/Moving*
>> --  Address         Load       Tokens  Owns    Host ID
>>             Rack
>> UN  162.243.86.41   1.08 MB    1       0.1%
>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>> UL  162.243.109.94  1.28 MB    256     99.9%
>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>>
>> U = Up, D = Down
>> N = Normal, L = Leaving, J = Joining and M = Moving
>>
>> On Mon, Oct 27, 2014 at 2:42 PM, Tim Dunphy <bl...@gmail.com> wrote:
>>
>>> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>>>> causing the problem
>>>
>>>
>>> OK, that's an interesting observation.How do you fix a node that is an
>>> UL state? What causes this?
>>>
>>> Also, is there any document that explains what all the nodetool
>>> abbreviations (UN, UL) stand for?
>>>
>>> On Mon, Oct 27, 2014 at 5:46 AM, jivko donev <ji...@yahoo.com>
>>> wrote:
>>>
>>>> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>>>> causing the problem.
>>>>
>>>>
>>>>   On Sunday, October 26, 2014 11:57 PM, Tim Dunphy <
>>>> bluethundr@gmail.com> wrote:
>>>>
>>>>
>>>> Hey all,
>>>>
>>>>  I'm trying to decommission a node.
>>>>
>>>>  First I'm getting a status:
>>>>
>>>> [root@beta-new:/usr/local] #nodetool status
>>>> Note: Ownership information does not include topology; for complete
>>>> information, specify a keyspace
>>>> Datacenter: datacenter1
>>>> =======================
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> --  Address         Load       Tokens  Owns    Host ID
>>>>               Rack
>>>> UN  162.243.86.41   1.08 MB    1       0.1%
>>>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>>>> UL  162.243.109.94  1.28 MB    256     99.9%
>>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>>>>
>>>>
>>>> But when I try to decommission the node I get this message:
>>>>
>>>> [root@beta-new:/usr/local] #nodetool -h 162.243.86.41 decommission
>>>> nodetool: Failed to connect to '162.243.86.41:7199' -
>>>> NoSuchObjectException: 'no such object in table'.
>>>>
>>>> Yet I can telnet to that host on that port just fine:
>>>>
>>>> [root@beta-new:/usr/local] #telnet 162.243.86.41 7199
>>>> Trying 162.243.86.41...
>>>> Connected to 162.243.86.41.
>>>> Escape character is '^]'.
>>>>
>>>>
>>>> And I have verified that cassandra is running and accessible via cqlsh
>>>> on the other machine.
>>>>
>>>> What could be going wrong?
>>>>
>>>> Thanks
>>>> Tim
>>>>
>>>>
>>>> --
>>>> GPG me!!
>>>>
>>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> GPG me!!
>>>
>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>
>>>
>>
>
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>
>

Re: decommissioning a cassandra node

Posted by Tim Dunphy <bl...@gmail.com>.
>
> "Also, is there any document that explains what all the nodetool
> abbreviations (UN, UL) stand for?"
> --> The documentation is in the command output itself
> Datacenter: datacenter1
> =======================
>
> *Status=Up/Down*
> *|/ State=Normal/Leaving/Joining/Moving*--  Address         Load
> Tokens  Owns    Host ID                               Rack
> UN  162.243.86.41   1.08 MB    1       0.1%
>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
> UL  162.243.109.94  1.28 MB    256     99.9%
> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
> U = Up, D = Down
> N = Normal, L = Leaving, J = Joining and M = Moving


Ok, got it, thanks!

Can someone suggest a good way to fix a node that is in an UL state?

Thanks
Tim

On Mon, Oct 27, 2014 at 9:46 AM, DuyHai Doan <do...@gmail.com> wrote:

> "Also, is there any document that explains what all the nodetool
> abbreviations (UN, UL) stand for?"
>
> --> The documentation is in the command output itself
>
> Datacenter: datacenter1
> =======================
> *Status=Up/Down*
> *|/ State=Normal/Leaving/Joining/Moving*
> --  Address         Load       Tokens  Owns    Host ID
>           Rack
> UN  162.243.86.41   1.08 MB    1       0.1%
>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
> UL  162.243.109.94  1.28 MB    256     99.9%
> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>
> U = Up, D = Down
> N = Normal, L = Leaving, J = Joining and M = Moving
>
> On Mon, Oct 27, 2014 at 2:42 PM, Tim Dunphy <bl...@gmail.com> wrote:
>
>> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>>> causing the problem
>>
>>
>> OK, that's an interesting observation.How do you fix a node that is an UL
>> state? What causes this?
>>
>> Also, is there any document that explains what all the nodetool
>> abbreviations (UN, UL) stand for?
>>
>> On Mon, Oct 27, 2014 at 5:46 AM, jivko donev <ji...@yahoo.com> wrote:
>>
>>> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>>> causing the problem.
>>>
>>>
>>>   On Sunday, October 26, 2014 11:57 PM, Tim Dunphy <bl...@gmail.com>
>>> wrote:
>>>
>>>
>>> Hey all,
>>>
>>>  I'm trying to decommission a node.
>>>
>>>  First I'm getting a status:
>>>
>>> [root@beta-new:/usr/local] #nodetool status
>>> Note: Ownership information does not include topology; for complete
>>> information, specify a keyspace
>>> Datacenter: datacenter1
>>> =======================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address         Load       Tokens  Owns    Host ID
>>>             Rack
>>> UN  162.243.86.41   1.08 MB    1       0.1%
>>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>>> UL  162.243.109.94  1.28 MB    256     99.9%
>>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>>>
>>>
>>> But when I try to decommission the node I get this message:
>>>
>>> [root@beta-new:/usr/local] #nodetool -h 162.243.86.41 decommission
>>> nodetool: Failed to connect to '162.243.86.41:7199' -
>>> NoSuchObjectException: 'no such object in table'.
>>>
>>> Yet I can telnet to that host on that port just fine:
>>>
>>> [root@beta-new:/usr/local] #telnet 162.243.86.41 7199
>>> Trying 162.243.86.41...
>>> Connected to 162.243.86.41.
>>> Escape character is '^]'.
>>>
>>>
>>> And I have verified that cassandra is running and accessible via cqlsh
>>> on the other machine.
>>>
>>> What could be going wrong?
>>>
>>> Thanks
>>> Tim
>>>
>>>
>>> --
>>> GPG me!!
>>>
>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>
>>>
>>>
>>>
>>
>>
>> --
>> GPG me!!
>>
>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>
>>
>


-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: decommissioning a cassandra node

Posted by DuyHai Doan <do...@gmail.com>.
"Also, is there any document that explains what all the nodetool
abbreviations (UN, UL) stand for?"

--> The documentation is in the command output itself

Datacenter: datacenter1
=======================
*Status=Up/Down*
*|/ State=Normal/Leaving/Joining/Moving*
--  Address         Load       Tokens  Owns    Host ID
          Rack
UN  162.243.86.41   1.08 MB    1       0.1%
 e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
UL  162.243.109.94  1.28 MB    256     99.9%
fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1

U = Up, D = Down
N = Normal, L = Leaving, J = Joining and M = Moving

On Mon, Oct 27, 2014 at 2:42 PM, Tim Dunphy <bl...@gmail.com> wrote:

> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>> causing the problem
>
>
> OK, that's an interesting observation.How do you fix a node that is an UL
> state? What causes this?
>
> Also, is there any document that explains what all the nodetool
> abbreviations (UN, UL) stand for?
>
> On Mon, Oct 27, 2014 at 5:46 AM, jivko donev <ji...@yahoo.com> wrote:
>
>> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
>> causing the problem.
>>
>>
>>   On Sunday, October 26, 2014 11:57 PM, Tim Dunphy <bl...@gmail.com>
>> wrote:
>>
>>
>> Hey all,
>>
>>  I'm trying to decommission a node.
>>
>>  First I'm getting a status:
>>
>> [root@beta-new:/usr/local] #nodetool status
>> Note: Ownership information does not include topology; for complete
>> information, specify a keyspace
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address         Load       Tokens  Owns    Host ID
>>             Rack
>> UN  162.243.86.41   1.08 MB    1       0.1%
>>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
>> UL  162.243.109.94  1.28 MB    256     99.9%
>> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>>
>>
>> But when I try to decommission the node I get this message:
>>
>> [root@beta-new:/usr/local] #nodetool -h 162.243.86.41 decommission
>> nodetool: Failed to connect to '162.243.86.41:7199' -
>> NoSuchObjectException: 'no such object in table'.
>>
>> Yet I can telnet to that host on that port just fine:
>>
>> [root@beta-new:/usr/local] #telnet 162.243.86.41 7199
>> Trying 162.243.86.41...
>> Connected to 162.243.86.41.
>> Escape character is '^]'.
>>
>>
>> And I have verified that cassandra is running and accessible via cqlsh on
>> the other machine.
>>
>> What could be going wrong?
>>
>> Thanks
>> Tim
>>
>>
>> --
>> GPG me!!
>>
>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>
>>
>>
>>
>
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>
>

Re: decommissioning a cassandra node

Posted by Tim Dunphy <bl...@gmail.com>.
>
> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
> causing the problem


OK, that's an interesting observation.How do you fix a node that is an UL
state? What causes this?

Also, is there any document that explains what all the nodetool
abbreviations (UN, UL) stand for?

On Mon, Oct 27, 2014 at 5:46 AM, jivko donev <ji...@yahoo.com> wrote:

> As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is
> causing the problem.
>
>
>   On Sunday, October 26, 2014 11:57 PM, Tim Dunphy <bl...@gmail.com>
> wrote:
>
>
> Hey all,
>
>  I'm trying to decommission a node.
>
>  First I'm getting a status:
>
> [root@beta-new:/usr/local] #nodetool status
> Note: Ownership information does not include topology; for complete
> information, specify a keyspace
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address         Load       Tokens  Owns    Host ID
>           Rack
> UN  162.243.86.41   1.08 MB    1       0.1%
>  e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1
> UL  162.243.109.94  1.28 MB    256     99.9%
> fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1
>
>
> But when I try to decommission the node I get this message:
>
> [root@beta-new:/usr/local] #nodetool -h 162.243.86.41 decommission
> nodetool: Failed to connect to '162.243.86.41:7199' -
> NoSuchObjectException: 'no such object in table'.
>
> Yet I can telnet to that host on that port just fine:
>
> [root@beta-new:/usr/local] #telnet 162.243.86.41 7199
> Trying 162.243.86.41...
> Connected to 162.243.86.41.
> Escape character is '^]'.
>
>
> And I have verified that cassandra is running and accessible via cqlsh on
> the other machine.
>
> What could be going wrong?
>
> Thanks
> Tim
>
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>
>
>
>


-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Re: decommissioning a cassandra node

Posted by jivko donev <ji...@yahoo.com>.
As I see the state 162.243.109.94 is UL(Up/Leaving) so maybe this is causing the problem. 

     On Sunday, October 26, 2014 11:57 PM, Tim Dunphy <bl...@gmail.com> wrote:
   

 Hey all,
 I'm trying to decommission a node. 
 First I'm getting a status:
[root@beta-new:/usr/local] #nodetool statusNote: Ownership information does not include topology; for complete information, specify a keyspaceDatacenter: datacenter1=======================Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address         Load       Tokens  Owns    Host ID                               RackUN  162.243.86.41   1.08 MB    1       0.1%    e945f3b5-2e3e-4a20-b1bd-e30c474a7634  rack1UL  162.243.109.94  1.28 MB    256     99.9%   fd2f76ae-8dcf-4e93-a37f-bf1e9088696e  rack1

But when I try to decommission the node I get this message:
[root@beta-new:/usr/local] #nodetool -h 162.243.86.41 decommissionnodetool: Failed to connect to '162.243.86.41:7199' - NoSuchObjectException: 'no such object in table'.
Yet I can telnet to that host on that port just fine:
[root@beta-new:/usr/local] #telnet 162.243.86.41 7199Trying 162.243.86.41...Connected to 162.243.86.41.Escape character is '^]'.

And I have verified that cassandra is running and accessible via cqlsh on the other machine. 
What could be going wrong? 
ThanksTim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B