You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Tom van den Berge <to...@gmail.com> on 2015/10/14 13:26:51 UTC
Removed node is not completely removed
I have removed a node with nodetool removenode, which completed ok.
Nodetool status does not list the node anymore.
But since then, Im seeing messages in my other nodes log files referring to
the removed node:
INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
InetAddress /10.68.56.200 is now DOWN
INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
1891) Removing tokens [85070591730234615865843651857942052863] for /
10.68.56.200
These two messages appear every minute.
I've tried nodetool removenode again (Host ID not found) and removenode
force (no token removals in process).
The jmx unsafeAssassinateEndpoint gives a NullPointerException.
What can I do to remove the node entirely?
Re: Removed node is not completely removed
Posted by Tom van den Berge <to...@gmail.com>.
Thanks Sebastian, a restart solved the problem!
On Wed, Oct 14, 2015 at 3:46 PM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:
> We still keep endpoints in memory. Not sure how you git to this state but
> try a rolling restart.
> On Oct 14, 2015 9:43 AM, "Tom van den Berge" <to...@gmail.com>
> wrote:
>
>> Thanks for that Michael, I did not know that. However, the node is not
>> listed in the system.peers table on any node, so it seems that the problem
>> is not in this table.
>>
>>
>>
>> On Wed, Oct 14, 2015 at 3:30 PM, Laing, Michael <
>> michael.laing@nytimes.com> wrote:
>>
>>> Remember that the system keyspace uses LocalStrategy: each node has its
>>> own set of system tables. -ml
>>>
>>> On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
>>> tom.vandenberge@gmail.com> wrote:
>>>
>>>> Hi Carlos,
>>>>
>>>> I'm using 2.1.6. The mysterious node is not in the peers table. Any
>>>> other ideas?
>>>> One of my existing nodes is not present in the system.peers table,
>>>> though. Should I be worried?
>>>>
>>>> Regards,
>>>> Tom
>>>>
>>>> On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>>>
>>>>> Check system.peers table to see if the IP is still there. If so edit
>>>>> the table and remove the offending IP.
>>>>>
>>>>> You are probably running into this:
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-6053
>>>>>
>>>>> Regards,
>>>>>
>>>>> Carlos Juzarte Rolo
>>>>> Cassandra Consultant
>>>>>
>>>>> Pythian - Love your data
>>>>>
>>>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>>>> www.pythian.com
>>>>>
>>>>> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
>>>>> tom.vandenberge@gmail.com> wrote:
>>>>>
>>>>>> I have removed a node with nodetool removenode, which completed ok.
>>>>>> Nodetool status does not list the node anymore.
>>>>>>
>>>>>> But since then, Im seeing messages in my other nodes log files
>>>>>> referring to the removed node:
>>>>>>
>>>>>> INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line
>>>>>> 968) InetAddress /10.68.56.200 is now DOWN
>>>>>> INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java
>>>>>> (line 1891) Removing tokens [85070591730234615865843651857942052863] for /
>>>>>> 10.68.56.200
>>>>>>
>>>>>>
>>>>>> These two messages appear every minute.
>>>>>> I've tried nodetool removenode again (Host ID not found) and
>>>>>> removenode force (no token removals in process).
>>>>>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>>>>>
>>>>>> What can I do to remove the node entirely?
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Removed node is not completely removed
Posted by Sebastian Estevez <se...@datastax.com>.
We still keep endpoints in memory. Not sure how you git to this state but
try a rolling restart.
On Oct 14, 2015 9:43 AM, "Tom van den Berge" <to...@gmail.com>
wrote:
> Thanks for that Michael, I did not know that. However, the node is not
> listed in the system.peers table on any node, so it seems that the problem
> is not in this table.
>
>
>
> On Wed, Oct 14, 2015 at 3:30 PM, Laing, Michael <michael.laing@nytimes.com
> > wrote:
>
>> Remember that the system keyspace uses LocalStrategy: each node has its
>> own set of system tables. -ml
>>
>> On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
>> tom.vandenberge@gmail.com> wrote:
>>
>>> Hi Carlos,
>>>
>>> I'm using 2.1.6. The mysterious node is not in the peers table. Any
>>> other ideas?
>>> One of my existing nodes is not present in the system.peers table,
>>> though. Should I be worried?
>>>
>>> Regards,
>>> Tom
>>>
>>> On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>>
>>>> Check system.peers table to see if the IP is still there. If so edit
>>>> the table and remove the offending IP.
>>>>
>>>> You are probably running into this:
>>>> https://issues.apache.org/jira/browse/CASSANDRA-6053
>>>>
>>>> Regards,
>>>>
>>>> Carlos Juzarte Rolo
>>>> Cassandra Consultant
>>>>
>>>> Pythian - Love your data
>>>>
>>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>>> www.pythian.com
>>>>
>>>> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
>>>> tom.vandenberge@gmail.com> wrote:
>>>>
>>>>> I have removed a node with nodetool removenode, which completed ok.
>>>>> Nodetool status does not list the node anymore.
>>>>>
>>>>> But since then, Im seeing messages in my other nodes log files
>>>>> referring to the removed node:
>>>>>
>>>>> INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line
>>>>> 968) InetAddress /10.68.56.200 is now DOWN
>>>>> INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java
>>>>> (line 1891) Removing tokens [85070591730234615865843651857942052863] for /
>>>>> 10.68.56.200
>>>>>
>>>>>
>>>>> These two messages appear every minute.
>>>>> I've tried nodetool removenode again (Host ID not found) and
>>>>> removenode force (no token removals in process).
>>>>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>>>>
>>>>> What can I do to remove the node entirely?
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Removed node is not completely removed
Posted by Tom van den Berge <to...@gmail.com>.
Thanks for that Michael, I did not know that. However, the node is not
listed in the system.peers table on any node, so it seems that the problem
is not in this table.
On Wed, Oct 14, 2015 at 3:30 PM, Laing, Michael <mi...@nytimes.com>
wrote:
> Remember that the system keyspace uses LocalStrategy: each node has its
> own set of system tables. -ml
>
> On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
> tom.vandenberge@gmail.com> wrote:
>
>> Hi Carlos,
>>
>> I'm using 2.1.6. The mysterious node is not in the peers table. Any other
>> ideas?
>> One of my existing nodes is not present in the system.peers table,
>> though. Should I be worried?
>>
>> Regards,
>> Tom
>>
>> On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo <ro...@pythian.com> wrote:
>>
>>> Check system.peers table to see if the IP is still there. If so edit the
>>> table and remove the offending IP.
>>>
>>> You are probably running into this:
>>> https://issues.apache.org/jira/browse/CASSANDRA-6053
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>>> www.pythian.com
>>>
>>> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
>>> tom.vandenberge@gmail.com> wrote:
>>>
>>>> I have removed a node with nodetool removenode, which completed ok.
>>>> Nodetool status does not list the node anymore.
>>>>
>>>> But since then, Im seeing messages in my other nodes log files
>>>> referring to the removed node:
>>>>
>>>> INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
>>>> InetAddress /10.68.56.200 is now DOWN
>>>> INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java
>>>> (line 1891) Removing tokens [85070591730234615865843651857942052863] for /
>>>> 10.68.56.200
>>>>
>>>>
>>>> These two messages appear every minute.
>>>> I've tried nodetool removenode again (Host ID not found) and removenode
>>>> force (no token removals in process).
>>>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>>>
>>>> What can I do to remove the node entirely?
>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>
>
Re: Removed node is not completely removed
Posted by "Laing, Michael" <mi...@nytimes.com>.
Remember that the system keyspace uses LocalStrategy: each node has its own
set of system tables. -ml
On Wed, Oct 14, 2015 at 9:17 AM, Tom van den Berge <
tom.vandenberge@gmail.com> wrote:
> Hi Carlos,
>
> I'm using 2.1.6. The mysterious node is not in the peers table. Any other
> ideas?
> One of my existing nodes is not present in the system.peers table, though.
> Should I be worried?
>
> Regards,
> Tom
>
> On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo <ro...@pythian.com> wrote:
>
>> Check system.peers table to see if the IP is still there. If so edit the
>> table and remove the offending IP.
>>
>> You are probably running into this:
>> https://issues.apache.org/jira/browse/CASSANDRA-6053
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
>> <http://linkedin.com/in/carlosjuzarterolo>*
>> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
>> www.pythian.com
>>
>> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
>> tom.vandenberge@gmail.com> wrote:
>>
>>> I have removed a node with nodetool removenode, which completed ok.
>>> Nodetool status does not list the node anymore.
>>>
>>> But since then, Im seeing messages in my other nodes log files referring
>>> to the removed node:
>>>
>>> INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
>>> InetAddress /10.68.56.200 is now DOWN
>>> INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
>>> 1891) Removing tokens [85070591730234615865843651857942052863] for /
>>> 10.68.56.200
>>>
>>>
>>> These two messages appear every minute.
>>> I've tried nodetool removenode again (Host ID not found) and removenode
>>> force (no token removals in process).
>>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>>
>>> What can I do to remove the node entirely?
>>>
>>>
>>>
>>
>> --
>>
>>
>>
>>
>
Re: Removed node is not completely removed
Posted by Tom van den Berge <to...@gmail.com>.
Hi Carlos,
I'm using 2.1.6. The mysterious node is not in the peers table. Any other
ideas?
One of my existing nodes is not present in the system.peers table, though.
Should I be worried?
Regards,
Tom
On Wed, Oct 14, 2015 at 2:27 PM, Carlos Rolo <ro...@pythian.com> wrote:
> Check system.peers table to see if the IP is still there. If so edit the
> table and remove the offending IP.
>
> You are probably running into this:
> https://issues.apache.org/jira/browse/CASSANDRA-6053
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
> tom.vandenberge@gmail.com> wrote:
>
>> I have removed a node with nodetool removenode, which completed ok.
>> Nodetool status does not list the node anymore.
>>
>> But since then, Im seeing messages in my other nodes log files referring
>> to the removed node:
>>
>> INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
>> InetAddress /10.68.56.200 is now DOWN
>> INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
>> 1891) Removing tokens [85070591730234615865843651857942052863] for /
>> 10.68.56.200
>>
>>
>> These two messages appear every minute.
>> I've tried nodetool removenode again (Host ID not found) and removenode
>> force (no token removals in process).
>> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>>
>> What can I do to remove the node entirely?
>>
>>
>>
>
> --
>
>
>
>
Re: Removed node is not completely removed
Posted by Carlos Rolo <ro...@pythian.com>.
Check system.peers table to see if the IP is still there. If so edit the
table and remove the offending IP.
You are probably running into this:
https://issues.apache.org/jira/browse/CASSANDRA-6053
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com
On Wed, Oct 14, 2015 at 12:26 PM, Tom van den Berge <
tom.vandenberge@gmail.com> wrote:
> I have removed a node with nodetool removenode, which completed ok.
> Nodetool status does not list the node anymore.
>
> But since then, Im seeing messages in my other nodes log files referring
> to the removed node:
>
> INFO [GossipStage:38] 2015-10-14 11:18:26,322 Gossiper.java (line 968)
> InetAddress /10.68.56.200 is now DOWN
> INFO [GossipStage:38] 2015-10-14 11:18:26,324 StorageService.java (line
> 1891) Removing tokens [85070591730234615865843651857942052863] for /
> 10.68.56.200
>
>
> These two messages appear every minute.
> I've tried nodetool removenode again (Host ID not found) and removenode
> force (no token removals in process).
> The jmx unsafeAssassinateEndpoint gives a NullPointerException.
>
> What can I do to remove the node entirely?
>
>
>
--
--