You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Roger Brown <ro...@perfectsearchcorp.com> on 2018/01/25 16:50:43 UTC
Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
My first thought is restart cassandra and monitor its log to make sure it
starts up.
On Thu, Jan 25, 2018 at 9:15 AM Jeff Jirsa <jj...@gmail.com> wrote:
> Can you telnet to the 7199 port?
>
> --
> Jeff Jirsa
>
>
> On Jan 25, 2018, at 8:00 AM, James Lovato <Ja...@iacapps.com>
> wrote:
>
> I have a 15 node cluster, 5 in each of 3 dcs. One host just recently
> started hanging doing any `nodetool` requests. I’ve enabled remote JMX on
> all these hosts and it was working fine until this week. No other changes
> have been made. All the Cassandra-env.sh files are the same among the
> nodes, and only seeds and listen_address are different for Cassandra.yaml
> files.
>
>
>
> Nothing in debug log about why the Connection timed out. This is what I
> get running nodetool from the same host:
>
>
>
> [root@host5 conf]# nodetool status
>
> nodetool: Failed to connect to 'xx.xx.xx.xx:7199' - ConnectException:
> 'Connection timed out (Connection timed out)'.
>
>
>
> Any thoughts or how to log nodetool’s actions? Or see why it’s not
> accepting the request? Telnet to the port from localhost as well as
> another node work fine.
>
>
>
> Thanks,
> James
>
>
Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
Posted by James Lovato <Ja...@iacapps.com>.
I’ve resolved the issue. These hosts were initially created in a different VLAN. There was a holdover entry in /etc/hosts pointing to the old IP Addr for this host.
So, the return back to the client never happened because it was sending the connection to the old IP to a host that did not exist.
Thanks for looking!
James
From: James Lovato <Ja...@iacapps.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Thursday, January 25, 2018 at 10:44 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
It starts up, I can telnet to the port. Also the log states so:
^C[root@dfprdbcdlpcs5 logs]# grep JMX cassandra.log
INFO [main] 2018-01-25 10:52:54,176 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO [main] 2018-01-25 10:52:54,379 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199
INFO [main] 2018-01-25 10:55:25,751 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO [main] 2018-01-25 10:55:25,957 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199
From: Roger Brown <ro...@perfectsearchcorp.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Thursday, January 25, 2018 at 9:51 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
My first thought is restart cassandra and monitor its log to make sure it starts up.
On Thu, Jan 25, 2018 at 9:15 AM Jeff Jirsa <jj...@gmail.com>> wrote:
Can you telnet to the 7199 port?
--
Jeff Jirsa
On Jan 25, 2018, at 8:00 AM, James Lovato <Ja...@iacapps.com>> wrote:
I have a 15 node cluster, 5 in each of 3 dcs. One host just recently started hanging doing any `nodetool` requests. I’ve enabled remote JMX on all these hosts and it was working fine until this week. No other changes have been made. All the Cassandra-env.sh files are the same among the nodes, and only seeds and listen_address are different for Cassandra.yaml files.
Nothing in debug log about why the Connection timed out. This is what I get running nodetool from the same host:
[root@host5 conf]# nodetool status
nodetool: Failed to connect to 'xx.xx.xx.xx:7199' - ConnectException: 'Connection timed out (Connection timed out)'.
Any thoughts or how to log nodetool’s actions? Or see why it’s not accepting the request? Telnet to the port from localhost as well as another node work fine.
Thanks,
James
Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
Posted by James Lovato <Ja...@iacapps.com>.
It starts up, I can telnet to the port. Also the log states so:
^C[root@dfprdbcdlpcs5 logs]# grep JMX cassandra.log
INFO [main] 2018-01-25 10:52:54,176 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO [main] 2018-01-25 10:52:54,379 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199
INFO [main] 2018-01-25 10:55:25,751 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi
INFO [main] 2018-01-25 10:55:25,957 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199
From: Roger Brown <ro...@perfectsearchcorp.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Thursday, January 25, 2018 at 9:51 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
My first thought is restart cassandra and monitor its log to make sure it starts up.
On Thu, Jan 25, 2018 at 9:15 AM Jeff Jirsa <jj...@gmail.com>> wrote:
Can you telnet to the 7199 port?
--
Jeff Jirsa
On Jan 25, 2018, at 8:00 AM, James Lovato <Ja...@iacapps.com>> wrote:
I have a 15 node cluster, 5 in each of 3 dcs. One host just recently started hanging doing any `nodetool` requests. I’ve enabled remote JMX on all these hosts and it was working fine until this week. No other changes have been made. All the Cassandra-env.sh files are the same among the nodes, and only seeds and listen_address are different for Cassandra.yaml files.
Nothing in debug log about why the Connection timed out. This is what I get running nodetool from the same host:
[root@host5 conf]# nodetool status
nodetool: Failed to connect to 'xx.xx.xx.xx:7199' - ConnectException: 'Connection timed out (Connection timed out)'.
Any thoughts or how to log nodetool’s actions? Or see why it’s not accepting the request? Telnet to the port from localhost as well as another node work fine.
Thanks,
James