You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rahul Reddy <ra...@gmail.com> on 2019/07/29 11:20:52 UTC
Jmx metrics shows node down
Hello,
Decommissioned 2 nodes from cluster nodetool status doesn't list the nodes
as expected but jmx metrics shows still those 2 nodes has down. Nodetool
gossip shows the 2 nodes in Left state. Why does my jmx still shows those
nodes down even after 24 hours. Cassandra version 3.11.3 ? Anything else
need to be done?
Re: Jmx metrics shows node down
Posted by yuping wang <yu...@gmail.com>.
We have the same issue. We observed the JMX only cleared after exactly 72 hours too.
On Jul 29, 2019, at 11:23 AM, Rahul Reddy <ra...@gmail.com> wrote:
And also system.peers table doesn't have the information on old nodes only ghost nodes to be there in JMX
> On Mon, Jul 29, 2019, 7:39 AM Rahul Reddy <ra...@gmail.com> wrote:
> We removed many times nodes from a cluster but never seen the jmx metric down stay for 72 hours. So it has to be completely removed from gossip to show the metric as expected? This would be problem for using the metric to alert on call
>
>> On Mon, Jul 29, 2019, 7:28 AM Oleksandr Shulgin <ol...@zalando.de> wrote:
>>> On Mon, Jul 29, 2019 at 1:21 PM Rahul Reddy <ra...@gmail.com> wrote:
>>
>>>
>>> Decommissioned 2 nodes from cluster nodetool status doesn't list the nodes as expected but jmx metrics shows still those 2 nodes has down. Nodetool gossip shows the 2 nodes in Left state. Why does my jmx still shows those nodes down even after 24 hours. Cassandra version 3.11.3 ?
>>
>> AFAIK, the nodes are not removed from gossip for 72 hours by default.
>>
>>> Anything else need to be done?
>>
>> Wait another 48 hours? ;-)
>>
>> --
>> Alex
>>
Re: Jmx metrics shows node down
Posted by Rahul Reddy <ra...@gmail.com>.
And also system.peers table doesn't have the information on old nodes only
ghost nodes to be there in JMX
On Mon, Jul 29, 2019, 7:39 AM Rahul Reddy <ra...@gmail.com> wrote:
> We removed many times nodes from a cluster but never seen the jmx metric
> down stay for 72 hours. So it has to be completely removed from gossip to
> show the metric as expected? This would be problem for using the metric to
> alert on call
>
> On Mon, Jul 29, 2019, 7:28 AM Oleksandr Shulgin <
> oleksandr.shulgin@zalando.de> wrote:
>
>> On Mon, Jul 29, 2019 at 1:21 PM Rahul Reddy <ra...@gmail.com>
>> wrote:
>>
>>>
>>> Decommissioned 2 nodes from cluster nodetool status doesn't list the
>>> nodes as expected but jmx metrics shows still those 2 nodes has down.
>>> Nodetool gossip shows the 2 nodes in Left state. Why does my jmx still
>>> shows those nodes down even after 24 hours. Cassandra version 3.11.3 ?
>>>
>>
>> AFAIK, the nodes are not removed from gossip for 72 hours by default.
>>
>>
>>> Anything else need to be done?
>>>
>>
>> Wait another 48 hours? ;-)
>>
>> --
>> Alex
>>
>>
Re: Jmx metrics shows node down
Posted by Rahul Reddy <ra...@gmail.com>.
We removed many times nodes from a cluster but never seen the jmx metric
down stay for 72 hours. So it has to be completely removed from gossip to
show the metric as expected? This would be problem for using the metric to
alert on call
On Mon, Jul 29, 2019, 7:28 AM Oleksandr Shulgin <
oleksandr.shulgin@zalando.de> wrote:
> On Mon, Jul 29, 2019 at 1:21 PM Rahul Reddy <ra...@gmail.com>
> wrote:
>
>>
>> Decommissioned 2 nodes from cluster nodetool status doesn't list the
>> nodes as expected but jmx metrics shows still those 2 nodes has down.
>> Nodetool gossip shows the 2 nodes in Left state. Why does my jmx still
>> shows those nodes down even after 24 hours. Cassandra version 3.11.3 ?
>>
>
> AFAIK, the nodes are not removed from gossip for 72 hours by default.
>
>
>> Anything else need to be done?
>>
>
> Wait another 48 hours? ;-)
>
> --
> Alex
>
>
RE: Jmx metrics shows node down
Posted by "ZAIDI, ASAD A" <az...@att.com>.
Another way to purge gossip info from each node is to:
1. Gracefully stop cassandra i.e. nodetool drain; kill Casandra PID
2. Move/delete files from $DATADIR/system/peers/
3. Add JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false" in jvm.options file
4. Restart Cassandra service.
5. Repeat above steps on each nodes in dc/cluster.
6. Once gossip info is purged, remove jvm option added in step 3 and restart instance again.
Depending on cluster,load size you may get this done swiftly.
~Asad
From: yuping wang [mailto:yupingwyp1218@gmail.com]
Sent: Monday, July 29, 2019 10:56 AM
To: user@cassandra.apache.org
Subject: Re: Jmx metrics shows node down
Is there workaround to shorten 72 hours to something shorter?(you said by default, wondering if one can set a non-default value?)
Thanks,
Yuping
On Jul 29, 2019, at 7:28 AM, Oleksandr Shulgin <ol...@zalando.de>> wrote:
On Mon, Jul 29, 2019 at 1:21 PM Rahul Reddy <ra...@gmail.com>> wrote:
Decommissioned 2 nodes from cluster nodetool status doesn't list the nodes as expected but jmx metrics shows still those 2 nodes has down. Nodetool gossip shows the 2 nodes in Left state. Why does my jmx still shows those nodes down even after 24 hours. Cassandra version 3.11.3 ?
AFAIK, the nodes are not removed from gossip for 72 hours by default.
Anything else need to be done?
Wait another 48 hours? ;-)
--
Alex
Re: Jmx metrics shows node down
Posted by yuping wang <yu...@gmail.com>.
Is there workaround to shorten 72 hours to something shorter?(you said by default, wondering if one can set a non-default value?)
Thanks,
Yuping
On Jul 29, 2019, at 7:28 AM, Oleksandr Shulgin <ol...@zalando.de> wrote:
> On Mon, Jul 29, 2019 at 1:21 PM Rahul Reddy <ra...@gmail.com> wrote:
>
> Decommissioned 2 nodes from cluster nodetool status doesn't list the nodes as expected but jmx metrics shows still those 2 nodes has down. Nodetool gossip shows the 2 nodes in Left state. Why does my jmx still shows those nodes down even after 24 hours. Cassandra version 3.11.3 ?
AFAIK, the nodes are not removed from gossip for 72 hours by default.
> Anything else need to be done?
Wait another 48 hours? ;-)
--
Alex
Re: Jmx metrics shows node down
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Mon, Jul 29, 2019 at 1:21 PM Rahul Reddy <ra...@gmail.com>
wrote:
>
> Decommissioned 2 nodes from cluster nodetool status doesn't list the
> nodes as expected but jmx metrics shows still those 2 nodes has down.
> Nodetool gossip shows the 2 nodes in Left state. Why does my jmx still
> shows those nodes down even after 24 hours. Cassandra version 3.11.3 ?
>
AFAIK, the nodes are not removed from gossip for 72 hours by default.
> Anything else need to be done?
>
Wait another 48 hours? ;-)
--
Alex