You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jeff Williams <je...@wherethebitsroam.com> on 2015/07/01 09:23:14 UTC
Re: Decommissioned node still in Gossip
Thanks for the tip Aiman, but this node is not in the seed list anywhere.
Jeff
On 30 June 2015 at 18:16, Aiman Parvaiz <ai...@flipagram.com> wrote:
> I was having exactly the same issue with the same version, check your seed
> list and make sure it contains only the live nodes, I know that seeds are
> only read when cassandra starts but updating the seed list to live nodes
> and then doing a roiling restart fixed this issue for me.
> I hope this helps you.
>
> Thanks
>
> Sent from my iPhone
>
> On Jun 30, 2015, at 4:42 AM, Jeff Williams <je...@wherethebitsroam.com>
> wrote:
>
> Hi,
>
> I have a cluster which had 4 datacenters running 2.0.12. Last week one of
> the datacenters was decommissioned using nodetool decommission on each of
> the servers in turn. This seemed to work fine until one of the nodes
> started appearing in the logs of all of the remaining servers with messages
> like:
>
> INFO [GossipStage:3] 2015-06-30 11:22:39,189 Gossiper.java (line 924)
> InetAddress /172.29.8.8 is now DOWN
> INFO [GossipStage:3] 2015-06-30 11:22:39,190 StorageService.java (line
> 1773) Removing tokens [...] for /172.29.8.8
>
> These come up in the log every minute or two. I believe it may have
> re-appeared after a repair, but I'm not sure.
>
> The problem is that this node does not exist in "nodetool status",
> "nodetool gossipinfo" or in the system.peers table. So how can tell the
> cluster that this node is decommissioned?
>
> Regards,
> Jeff
>
>
Re: Decommissioned node still in Gossip
Posted by Jeff Williams <je...@wherethebitsroam.com>.
Solved. Finally found that there was one node with 172.29.8.8 in gossipinfo
and appearing incomplete in status:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UL 172.29.8.8 13.28 GB 256 9.5% null
1
Restarting this node removed the decommissioned node from gossip.
Jeff
On 3 July 2015 at 13:09, Jeff Williams <je...@wherethebitsroam.com> wrote:
> I'm thinking I'll have to find something similar for 2.0. I just don't
> understand where the node is coming from!
>
> Jeff
>
> On 1 July 2015 at 10:21, Vitalii Skakun <vi...@gmail.com> wrote:
>
>> Hi,
>>
>> just a guess, there was a possibility to purge gossip state on a node, at
>> least in version 1.2
>>
>> http://docs.datastax.com/en/cassandra/1.2/cassandra/architecture/architectureGossipPurge_t.html
>>
>> the trick was to add -Dcassandra.load_ring_state=false somehow to the
>> jvm parameters
>>
>> I'm not sure if it works for 2.0 (since it's not in the doc for 2.x)
>>
>> 2015-07-01 10:23 GMT+03:00 Jeff Williams <je...@wherethebitsroam.com>:
>>
>>> Thanks for the tip Aiman, but this node is not in the seed list anywhere.
>>>
>>> Jeff
>>>
>>> On 30 June 2015 at 18:16, Aiman Parvaiz <ai...@flipagram.com> wrote:
>>>
>>>> I was having exactly the same issue with the same version, check your
>>>> seed list and make sure it contains only the live nodes, I know that seeds
>>>> are only read when cassandra starts but updating the seed list to live
>>>> nodes and then doing a roiling restart fixed this issue for me.
>>>> I hope this helps you.
>>>>
>>>> Thanks
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 30, 2015, at 4:42 AM, Jeff Williams <je...@wherethebitsroam.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a cluster which had 4 datacenters running 2.0.12. Last week one
>>>> of the datacenters was decommissioned using nodetool decommission on each
>>>> of the servers in turn. This seemed to work fine until one of the nodes
>>>> started appearing in the logs of all of the remaining servers with messages
>>>> like:
>>>>
>>>> INFO [GossipStage:3] 2015-06-30 11:22:39,189 Gossiper.java (line 924)
>>>> InetAddress /172.29.8.8 is now DOWN
>>>> INFO [GossipStage:3] 2015-06-30 11:22:39,190 StorageService.java (line
>>>> 1773) Removing tokens [...] for /172.29.8.8
>>>>
>>>> These come up in the log every minute or two. I believe it may have
>>>> re-appeared after a repair, but I'm not sure.
>>>>
>>>> The problem is that this node does not exist in "nodetool status",
>>>> "nodetool gossipinfo" or in the system.peers table. So how can tell the
>>>> cluster that this node is decommissioned?
>>>>
>>>> Regards,
>>>> Jeff
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> ----
>> with best regards,
>> Vitalii Skakun
>>
>
>
Re: Decommissioned node still in Gossip
Posted by Sebastian Estevez <se...@datastax.com>.
Is the ghost node in your system.peers on any node? Check every node (since
it's local strategy).
Otherwise the nodes must be in the endpoint list in memory and you'll have
to rolling restart. Make sure you drain though before the rolling restart
since it may be in your commit log somewhere too.
On Jul 3, 2015 7:11 AM, "Jeff Williams" <je...@wherethebitsroam.com> wrote:
> I'm thinking I'll have to find something similar for 2.0. I just don't
> understand where the node is coming from!
>
> Jeff
>
> On 1 July 2015 at 10:21, Vitalii Skakun <vi...@gmail.com> wrote:
>
>> Hi,
>>
>> just a guess, there was a possibility to purge gossip state on a node, at
>> least in version 1.2
>>
>> http://docs.datastax.com/en/cassandra/1.2/cassandra/architecture/architectureGossipPurge_t.html
>>
>> the trick was to add -Dcassandra.load_ring_state=false somehow to the
>> jvm parameters
>>
>> I'm not sure if it works for 2.0 (since it's not in the doc for 2.x)
>>
>> 2015-07-01 10:23 GMT+03:00 Jeff Williams <je...@wherethebitsroam.com>:
>>
>>> Thanks for the tip Aiman, but this node is not in the seed list anywhere.
>>>
>>> Jeff
>>>
>>> On 30 June 2015 at 18:16, Aiman Parvaiz <ai...@flipagram.com> wrote:
>>>
>>>> I was having exactly the same issue with the same version, check your
>>>> seed list and make sure it contains only the live nodes, I know that seeds
>>>> are only read when cassandra starts but updating the seed list to live
>>>> nodes and then doing a roiling restart fixed this issue for me.
>>>> I hope this helps you.
>>>>
>>>> Thanks
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 30, 2015, at 4:42 AM, Jeff Williams <je...@wherethebitsroam.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a cluster which had 4 datacenters running 2.0.12. Last week one
>>>> of the datacenters was decommissioned using nodetool decommission on each
>>>> of the servers in turn. This seemed to work fine until one of the nodes
>>>> started appearing in the logs of all of the remaining servers with messages
>>>> like:
>>>>
>>>> INFO [GossipStage:3] 2015-06-30 11:22:39,189 Gossiper.java (line 924)
>>>> InetAddress /172.29.8.8 is now DOWN
>>>> INFO [GossipStage:3] 2015-06-30 11:22:39,190 StorageService.java (line
>>>> 1773) Removing tokens [...] for /172.29.8.8
>>>>
>>>> These come up in the log every minute or two. I believe it may have
>>>> re-appeared after a repair, but I'm not sure.
>>>>
>>>> The problem is that this node does not exist in "nodetool status",
>>>> "nodetool gossipinfo" or in the system.peers table. So how can tell the
>>>> cluster that this node is decommissioned?
>>>>
>>>> Regards,
>>>> Jeff
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> ----
>> with best regards,
>> Vitalii Skakun
>>
>
>
Re: Decommissioned node still in Gossip
Posted by Jeff Williams <je...@wherethebitsroam.com>.
I'm thinking I'll have to find something similar for 2.0. I just don't
understand where the node is coming from!
Jeff
On 1 July 2015 at 10:21, Vitalii Skakun <vi...@gmail.com> wrote:
> Hi,
>
> just a guess, there was a possibility to purge gossip state on a node, at
> least in version 1.2
>
> http://docs.datastax.com/en/cassandra/1.2/cassandra/architecture/architectureGossipPurge_t.html
>
> the trick was to add -Dcassandra.load_ring_state=false somehow to the jvm
> parameters
>
> I'm not sure if it works for 2.0 (since it's not in the doc for 2.x)
>
> 2015-07-01 10:23 GMT+03:00 Jeff Williams <je...@wherethebitsroam.com>:
>
>> Thanks for the tip Aiman, but this node is not in the seed list anywhere.
>>
>> Jeff
>>
>> On 30 June 2015 at 18:16, Aiman Parvaiz <ai...@flipagram.com> wrote:
>>
>>> I was having exactly the same issue with the same version, check your
>>> seed list and make sure it contains only the live nodes, I know that seeds
>>> are only read when cassandra starts but updating the seed list to live
>>> nodes and then doing a roiling restart fixed this issue for me.
>>> I hope this helps you.
>>>
>>> Thanks
>>>
>>> Sent from my iPhone
>>>
>>> On Jun 30, 2015, at 4:42 AM, Jeff Williams <je...@wherethebitsroam.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I have a cluster which had 4 datacenters running 2.0.12. Last week one
>>> of the datacenters was decommissioned using nodetool decommission on each
>>> of the servers in turn. This seemed to work fine until one of the nodes
>>> started appearing in the logs of all of the remaining servers with messages
>>> like:
>>>
>>> INFO [GossipStage:3] 2015-06-30 11:22:39,189 Gossiper.java (line 924)
>>> InetAddress /172.29.8.8 is now DOWN
>>> INFO [GossipStage:3] 2015-06-30 11:22:39,190 StorageService.java (line
>>> 1773) Removing tokens [...] for /172.29.8.8
>>>
>>> These come up in the log every minute or two. I believe it may have
>>> re-appeared after a repair, but I'm not sure.
>>>
>>> The problem is that this node does not exist in "nodetool status",
>>> "nodetool gossipinfo" or in the system.peers table. So how can tell the
>>> cluster that this node is decommissioned?
>>>
>>> Regards,
>>> Jeff
>>>
>>>
>>
>
>
> --
>
> ----
> with best regards,
> Vitalii Skakun
>
Re: Decommissioned node still in Gossip
Posted by Vitalii Skakun <vi...@gmail.com>.
Hi,
just a guess, there was a possibility to purge gossip state on a node, at
least in version 1.2
http://docs.datastax.com/en/cassandra/1.2/cassandra/architecture/architectureGossipPurge_t.html
the trick was to add -Dcassandra.load_ring_state=false somehow to the jvm
parameters
I'm not sure if it works for 2.0 (since it's not in the doc for 2.x)
2015-07-01 10:23 GMT+03:00 Jeff Williams <je...@wherethebitsroam.com>:
> Thanks for the tip Aiman, but this node is not in the seed list anywhere.
>
> Jeff
>
> On 30 June 2015 at 18:16, Aiman Parvaiz <ai...@flipagram.com> wrote:
>
>> I was having exactly the same issue with the same version, check your
>> seed list and make sure it contains only the live nodes, I know that seeds
>> are only read when cassandra starts but updating the seed list to live
>> nodes and then doing a roiling restart fixed this issue for me.
>> I hope this helps you.
>>
>> Thanks
>>
>> Sent from my iPhone
>>
>> On Jun 30, 2015, at 4:42 AM, Jeff Williams <je...@wherethebitsroam.com>
>> wrote:
>>
>> Hi,
>>
>> I have a cluster which had 4 datacenters running 2.0.12. Last week one of
>> the datacenters was decommissioned using nodetool decommission on each of
>> the servers in turn. This seemed to work fine until one of the nodes
>> started appearing in the logs of all of the remaining servers with messages
>> like:
>>
>> INFO [GossipStage:3] 2015-06-30 11:22:39,189 Gossiper.java (line 924)
>> InetAddress /172.29.8.8 is now DOWN
>> INFO [GossipStage:3] 2015-06-30 11:22:39,190 StorageService.java (line
>> 1773) Removing tokens [...] for /172.29.8.8
>>
>> These come up in the log every minute or two. I believe it may have
>> re-appeared after a repair, but I'm not sure.
>>
>> The problem is that this node does not exist in "nodetool status",
>> "nodetool gossipinfo" or in the system.peers table. So how can tell the
>> cluster that this node is decommissioned?
>>
>> Regards,
>> Jeff
>>
>>
>
--
----
with best regards,
Vitalii Skakun