You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "R. Verlangen" <ro...@us2.nl> on 2012/03/07 09:11:35 UTC
Node joining / unknown
Hi there,
I'm currently in a really weird situation.
- Nodetool ring says node X is joining (this already takes 12 hours, with
no activity)
- When I try to remove the token, it says: Exception in thread "main"
java.lang.UnsupportedOperationException: Token not found.
- Removetoken status = No token removals in process.
How to get that node out of my cluster?
With kind regards,
Robin Verlangen
Re: Node joining / unknown
Posted by Brandon Williams <dr...@gmail.com>.
On Wed, Mar 7, 2012 at 3:37 AM, aaron morton <aa...@thelastpickle.com> wrote:
> 2) Stop the node. Try to get remove the token again from another node. Node
> that removing a token will stream data around the place as well.
A node that has never fully joined doesn't need to be removed (and
can't.) Just shut it down and it will go away after a minute or so.
-Brandon
Re: Node joining / unknown
Posted by "R. Verlangen" <ro...@us2.nl>.
It seemed that one of the other nodes had trouble with a compaction task.
The C node was waiting for that.
It's now streaming all it's data into place.
Thank you all for your time!
2012/3/7 <ig...@4friends.od.ua>
> just run "nodetool compactionstat" on other nodes.
>
>
> -----Original Message-----
> From: "R. Verlangen" <ro...@us2.nl>
> To: user@cassandra.apache.org
> Sent: Wed, 07 Mar 2012 23:09
> Subject: Re: Node joining / unknown
>
> @Brandon: Thank you for the information. I'll do that next time.
>
> @Igor: Any ways to find out whether that is the current state? And if so,
> how to solve it?
>
> 2012/3/7 <ig...@4friends.od.ua>
>
>> Maybe it wait for verification compaction on other node?
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: "R. Verlangen" <ro...@us2.nl>
>> To: user@cassandra.apache.org
>> Sent: Wed, 07 Mar 2012 22:15
>> Subject: Re: Node joining / unknown
>>
>> At this moment the node has joined the ring (after a restart: tried that
>> before, but now it had finally result).
>>
>> When I try to run repair on the new node, the log says (the new node is
>> NODE C):
>>
>> INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,453 AntiEntropyService.java
>> (line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
>> tree for StorageMeta from NODE A
>> INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,643
>> AntiEntropyService.java (line 190) [repair
>> #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle tree for StorageMeta
>> from NODE B
>>
>> And then doesn't do anything anymore. Tried it a couple of times again.
>> It's just not starting.
>>
>> Results from netstats on NODE C:
>>
>> Mode: NORMAL
>> Not sending any streams.
>> Not receiving any streams.
>> Pool Name Active Pending Completed
>> Commands n/a 0 5
>> Responses n/a 93 4296
>>
>>
>> Any suggestions?
>>
>> Thank you!
>>
>> 2012/3/7 aaron morton <aa...@thelastpickle.com>
>>
>>> - When I try to remove the token, it says: Exception in thread "main"
>>> java.lang.UnsupportedOperationException: Token not found.
>>>
>>> Am assuming you ran nodetool removetoken on a node other than the
>>> joining node? What did nodetool ring look like on that machine ?
>>>
>>> Take a look at nodetool netstats on the joining node to see if streaming
>>> has failed. If it's dead then…
>>>
>>> 1) Try restarting the joining node and run nodetool repair on it
>>> immediately. Note: am assuming QUOURM CL otherwise things may get
>>> inconsistent.
>>> or
>>> 2) Stop the node. Try to get remove the token again from another node.
>>> Node that removing a token will stream data around the place as well.
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 7/03/2012, at 9:11 PM, R. Verlangen wrote:
>>>
>>> Hi there,
>>>
>>> I'm currently in a really weird situation.
>>> - Nodetool ring says node X is joining (this already takes 12 hours,
>>> with no activity)
>>> - When I try to remove the token, it says: Exception in thread "main"
>>> java.lang.UnsupportedOperationException: Token not found.
>>> - Removetoken status = No token removals in process.
>>>
>>> How to get that node out of my cluster?
>>>
>>> With kind regards,
>>> Robin Verlangen
>>>
>>>
>>>
>>
>
Re: Node joining / unknown
Posted by ig...@4friends.od.ua.
just run "nodetool compactionstat" on other nodes.
-----Original Message-----
From: "R. Verlangen" <ro...@us2.nl>
To: user@cassandra.apache.org
Sent: Wed, 07 Mar 2012 23:09
Subject: Re: Node joining / unknown
@Brandon: Thank you for the information. I'll do that next time.
@Igor: Any ways to find out whether that is the current state? And if so,
how to solve it?
2012/3/7 <ig...@4friends.od.ua>
> Maybe it wait for verification compaction on other node?
>
>
>
>
>
> -----Original Message-----
> From: "R. Verlangen" <ro...@us2.nl>
> To: user@cassandra.apache.org
> Sent: Wed, 07 Mar 2012 22:15
> Subject: Re: Node joining / unknown
>
> At this moment the node has joined the ring (after a restart: tried that
> before, but now it had finally result).
>
> When I try to run repair on the new node, the log says (the new node is
> NODE C):
>
> INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,453 AntiEntropyService.java
> (line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
> tree for StorageMeta from NODE A
> INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,643 AntiEntropyService.java
> (line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
> tree for StorageMeta from NODE B
>
> And then doesn't do anything anymore. Tried it a couple of times again.
> It's just not starting.
>
> Results from netstats on NODE C:
>
> Mode: NORMAL
> Not sending any streams.
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 0 5
> Responses n/a 93 4296
>
>
> Any suggestions?
>
> Thank you!
>
> 2012/3/7 aaron morton <aa...@thelastpickle.com>
>
>> - When I try to remove the token, it says: Exception in thread "main"
>> java.lang.UnsupportedOperationException: Token not found.
>>
>> Am assuming you ran nodetool removetoken on a node other than the joining
>> node? What did nodetool ring look like on that machine ?
>>
>> Take a look at nodetool netstats on the joining node to see if streaming
>> has failed. If it's dead then…
>>
>> 1) Try restarting the joining node and run nodetool repair on it
>> immediately. Note: am assuming QUOURM CL otherwise things may get
>> inconsistent.
>> or
>> 2) Stop the node. Try to get remove the token again from another node.
>> Node that removing a token will stream data around the place as well.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7/03/2012, at 9:11 PM, R. Verlangen wrote:
>>
>> Hi there,
>>
>> I'm currently in a really weird situation.
>> - Nodetool ring says node X is joining (this already takes 12 hours, with
>> no activity)
>> - When I try to remove the token, it says: Exception in thread "main"
>> java.lang.UnsupportedOperationException: Token not found.
>> - Removetoken status = No token removals in process.
>>
>> How to get that node out of my cluster?
>>
>> With kind regards,
>> Robin Verlangen
>>
>>
>>
>
Re: Node joining / unknown
Posted by "R. Verlangen" <ro...@us2.nl>.
@Brandon: Thank you for the information. I'll do that next time.
@Igor: Any ways to find out whether that is the current state? And if so,
how to solve it?
2012/3/7 <ig...@4friends.od.ua>
> Maybe it wait for verification compaction on other node?
>
>
>
>
>
> -----Original Message-----
> From: "R. Verlangen" <ro...@us2.nl>
> To: user@cassandra.apache.org
> Sent: Wed, 07 Mar 2012 22:15
> Subject: Re: Node joining / unknown
>
> At this moment the node has joined the ring (after a restart: tried that
> before, but now it had finally result).
>
> When I try to run repair on the new node, the log says (the new node is
> NODE C):
>
> INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,453 AntiEntropyService.java
> (line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
> tree for StorageMeta from NODE A
> INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,643 AntiEntropyService.java
> (line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
> tree for StorageMeta from NODE B
>
> And then doesn't do anything anymore. Tried it a couple of times again.
> It's just not starting.
>
> Results from netstats on NODE C:
>
> Mode: NORMAL
> Not sending any streams.
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 0 5
> Responses n/a 93 4296
>
>
> Any suggestions?
>
> Thank you!
>
> 2012/3/7 aaron morton <aa...@thelastpickle.com>
>
>> - When I try to remove the token, it says: Exception in thread "main"
>> java.lang.UnsupportedOperationException: Token not found.
>>
>> Am assuming you ran nodetool removetoken on a node other than the joining
>> node? What did nodetool ring look like on that machine ?
>>
>> Take a look at nodetool netstats on the joining node to see if streaming
>> has failed. If it's dead then…
>>
>> 1) Try restarting the joining node and run nodetool repair on it
>> immediately. Note: am assuming QUOURM CL otherwise things may get
>> inconsistent.
>> or
>> 2) Stop the node. Try to get remove the token again from another node.
>> Node that removing a token will stream data around the place as well.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7/03/2012, at 9:11 PM, R. Verlangen wrote:
>>
>> Hi there,
>>
>> I'm currently in a really weird situation.
>> - Nodetool ring says node X is joining (this already takes 12 hours, with
>> no activity)
>> - When I try to remove the token, it says: Exception in thread "main"
>> java.lang.UnsupportedOperationException: Token not found.
>> - Removetoken status = No token removals in process.
>>
>> How to get that node out of my cluster?
>>
>> With kind regards,
>> Robin Verlangen
>>
>>
>>
>
Re: Node joining / unknown
Posted by ig...@4friends.od.ua.
Maybe it wait for verification compaction on other node?
-----Original Message-----
From: "R. Verlangen" <ro...@us2.nl>
To: user@cassandra.apache.org
Sent: Wed, 07 Mar 2012 22:15
Subject: Re: Node joining / unknown
At this moment the node has joined the ring (after a restart: tried that
before, but now it had finally result).
When I try to run repair on the new node, the log says (the new node is
NODE C):
INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,453 AntiEntropyService.java
(line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
tree for StorageMeta from NODE A
INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,643 AntiEntropyService.java
(line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
tree for StorageMeta from NODE B
And then doesn't do anything anymore. Tried it a couple of times again.
It's just not starting.
Results from netstats on NODE C:
Mode: NORMAL
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 5
Responses n/a 93 4296
Any suggestions?
Thank you!
2012/3/7 aaron morton <aa...@thelastpickle.com>
> - When I try to remove the token, it says: Exception in thread "main"
> java.lang.UnsupportedOperationException: Token not found.
>
> Am assuming you ran nodetool removetoken on a node other than the joining
> node? What did nodetool ring look like on that machine ?
>
> Take a look at nodetool netstats on the joining node to see if streaming
> has failed. If it's dead then…
>
> 1) Try restarting the joining node and run nodetool repair on it
> immediately. Note: am assuming QUOURM CL otherwise things may get
> inconsistent.
> or
> 2) Stop the node. Try to get remove the token again from another node.
> Node that removing a token will stream data around the place as well.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7/03/2012, at 9:11 PM, R. Verlangen wrote:
>
> Hi there,
>
> I'm currently in a really weird situation.
> - Nodetool ring says node X is joining (this already takes 12 hours, with
> no activity)
> - When I try to remove the token, it says: Exception in thread "main"
> java.lang.UnsupportedOperationException: Token not found.
> - Removetoken status = No token removals in process.
>
> How to get that node out of my cluster?
>
> With kind regards,
> Robin Verlangen
>
>
>
Re: Node joining / unknown
Posted by "R. Verlangen" <ro...@us2.nl>.
At this moment the node has joined the ring (after a restart: tried that
before, but now it had finally result).
When I try to run repair on the new node, the log says (the new node is
NODE C):
INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,453 AntiEntropyService.java
(line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
tree for StorageMeta from NODE A
INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,643 AntiEntropyService.java
(line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
tree for StorageMeta from NODE B
And then doesn't do anything anymore. Tried it a couple of times again.
It's just not starting.
Results from netstats on NODE C:
Mode: NORMAL
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 5
Responses n/a 93 4296
Any suggestions?
Thank you!
2012/3/7 aaron morton <aa...@thelastpickle.com>
> - When I try to remove the token, it says: Exception in thread "main"
> java.lang.UnsupportedOperationException: Token not found.
>
> Am assuming you ran nodetool removetoken on a node other than the joining
> node? What did nodetool ring look like on that machine ?
>
> Take a look at nodetool netstats on the joining node to see if streaming
> has failed. If it's dead then…
>
> 1) Try restarting the joining node and run nodetool repair on it
> immediately. Note: am assuming QUOURM CL otherwise things may get
> inconsistent.
> or
> 2) Stop the node. Try to get remove the token again from another node.
> Node that removing a token will stream data around the place as well.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7/03/2012, at 9:11 PM, R. Verlangen wrote:
>
> Hi there,
>
> I'm currently in a really weird situation.
> - Nodetool ring says node X is joining (this already takes 12 hours, with
> no activity)
> - When I try to remove the token, it says: Exception in thread "main"
> java.lang.UnsupportedOperationException: Token not found.
> - Removetoken status = No token removals in process.
>
> How to get that node out of my cluster?
>
> With kind regards,
> Robin Verlangen
>
>
>
Re: Node joining / unknown
Posted by aaron morton <aa...@thelastpickle.com>.
> - When I try to remove the token, it says: Exception in thread "main" java.lang.UnsupportedOperationException: Token not found.
Am assuming you ran nodetool removetoken on a node other than the joining node? What did nodetool ring look like on that machine ?
Take a look at nodetool netstats on the joining node to see if streaming has failed. If it's dead then…
1) Try restarting the joining node and run nodetool repair on it immediately. Note: am assuming QUOURM CL otherwise things may get inconsistent.
or
2) Stop the node. Try to get remove the token again from another node. Node that removing a token will stream data around the place as well.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/03/2012, at 9:11 PM, R. Verlangen wrote:
> Hi there,
>
> I'm currently in a really weird situation.
> - Nodetool ring says node X is joining (this already takes 12 hours, with no activity)
> - When I try to remove the token, it says: Exception in thread "main" java.lang.UnsupportedOperationException: Token not found.
> - Removetoken status = No token removals in process.
>
> How to get that node out of my cluster?
>
> With kind regards,
> Robin Verlangen