You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Sun, Guan" <gu...@amazon.com> on 2017/01/16 23:29:18 UTC
Unreliable JMX metrics
Hi all,
We have a 6 node Cassandra cluster running on Amazon EC2. We replaced the nodes one by one while Cassandra was running several days ago. After the replacement, the nodetool status show that all the nodes were up and running. However, the JMX metrics about down nodes kept showing there were 6 nodes down for about 2.5 days before it became 0.
Has anyone seen this before? Any idea why the JMX metrics takes so long to reflect the actual status of the cluster?
Thanks,
Guan
Re: Unreliable JMX metrics
Posted by kurt Greaves <ku...@instaclustr.com>.
Yes. You likely will still be able to see the nodes in nodetool gossipinfo
Re: Unreliable JMX metrics
Posted by "Sun, Guan" <gu...@amazon.com>.
Thanks for you reply Malte.
I replaced the nodes and the new nodes have new IP, but I have removed the dead nodes using nodetool. Does Cassandra still keep the info of dead nodes for 72 hours even I have removed them?
Thanks,
Guan
From: Malte Pickhan <ma...@zalando.de>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Tuesday, January 17, 2017 at 7:48 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Unreliable JMX metrics
Hey Guan,
If it's the case that you actually replaced the nodes and already assigned new IP's to them this is probably common behaviour. Since Cassandra has a retention policy to keep dead nodes for 72 hours.
Best,
Malte
2017-01-17 0:29 GMT+01:00 Sun, Guan <gu...@amazon.com>>:
Hi all,
We have a 6 node Cassandra cluster running on Amazon EC2. We replaced the nodes one by one while Cassandra was running several days ago. After the replacement, the nodetool status show that all the nodes were up and running. However, the JMX metrics about down nodes kept showing there were 6 nodes down for about 2.5 days before it became 0.
Has anyone seen this before? Any idea why the JMX metrics takes so long to reflect the actual status of the cluster?
Thanks,
Guan
--
Malte Pickhan
Software Engineer
Zalando Payments SE & Co. KG
POSTAL ADDRESS
Zalando Payments SE & Co. KG
11501 Berlin
OFFICE
Zalando Payments SE & Co. KG
Hörder Hafenstraße 1-3
44263 Dortmund
Germany
E-Mail: malte.pickhan@zalando.de<ma...@zalando.de>
Web: corporate.zalando.de<http://corporate.zalando.de/>
Jobs: jobs.zalando.de<http://jobs.zalando.de/>
Zalando Payments SE & Co. KG
Company registration: Amtsgericht Charlottenburg, HRA 48907 B
VAT registration number: DE 289683421
General partner: Zalando SE, registered office: Berlin
Company registration: Amtsgericht Charlottenburg, HRA 48907 B
Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of Supervisory: Lothar Lanz
Re: Unreliable JMX metrics
Posted by Malte Pickhan <ma...@zalando.de>.
Hey Guan,
If it's the case that you actually replaced the nodes and already assigned
new IP's to them this is probably common behaviour. Since Cassandra has a
retention policy to keep dead nodes for 72 hours.
Best,
Malte
2017-01-17 0:29 GMT+01:00 Sun, Guan <gu...@amazon.com>:
> Hi all,
>
> We have a 6 node Cassandra cluster running on Amazon EC2. We replaced the
> nodes one by one while Cassandra was running several days ago. After the
> replacement, the nodetool status show that all the nodes were up and
> running. However, the JMX metrics about down nodes kept showing there were
> 6 nodes down for about 2.5 days before it became 0.
>
> Has anyone seen this before? Any idea why the JMX metrics takes so long to
> reflect the actual status of the cluster?
>
> Thanks,
> Guan
>
--
Malte Pickhan
Software Engineer
Zalando Payments SE & Co. KG
*POSTAL ADDRESS*
Zalando Payments SE & Co. KG
11501 Berlin
*OFFICE*
Zalando Payments SE & Co. KG
Hörder Hafenstraße 1-3
44263 Dortmund
Germany
E-Mail: malte.pickhan@zalando.de
Web: *corporate.zalando.de <http://corporate.zalando.de/>*
Jobs: jobs.zalando.de
Zalando Payments SE & Co. KG
Company registration: Amtsgericht Charlottenburg, HRA 48907 B
VAT registration number: DE 289683421
General partner: Zalando SE, registered office: Berlin
Company registration: Amtsgericht Charlottenburg, HRA 48907 B
Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of Supervisory: Lothar Lanz