You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Arjan van Ham <ar...@bunchball.com> on 2010/12/14 09:36:36 UTC

Dual NIC server problems

I have a cluster with three nodes, version 0.7.0 RC2. Each node has dual NIC's, eth0 to the internet and eth1 to a private network (192.168.1.xxx). The outside NIC  on each node is frewalled using iptables, only port 22 is allowed through. My cassandra.yaml configuration file refers only to the internal hostnames that resolve to 192.168.1.xxx addresses.

Most of my Cassandra installation works fine, except for a few nodetool commands. It seems that they are trying to access the servers over the outside (firewalled) NIC and thus fail. If I disable the firewall they work fine. 

For example:

[cassandra@node11 ~]$ nodetool -h 192.168.1.153 decommission
Error connecting to remote JMX agent!
java.rmi.ConnectException: Connection refused to host: 72.52.199.70; nested exception is: 
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
        at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:115)
        at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:91)


And if I disable the firewall on the remote node it works right away.


Any ideas anyone?

Thanks.

Re: Dual NIC server problems

Posted by Arjan van Ham <ar...@bunchball.com>.
This  indeed fixed the problem.
Thanks Aaron and Oleg for your help in this.


On Dec 14, 2010, at 11:26 PM, Oleg Anastasyev wrote:

> This is probably because rmi code jmx uses to listen detected wrong address.
> To fix this add the following to cassandra nodes startup script instances:
> -Djava.rmi.server.hostname=127.0.0.1
> (change 127.0.0.1 to actual internal address of cassandra node)
>

Re: Dual NIC server problems

Posted by Oleg Anastasyev <ol...@gmail.com>.
This is probably because rmi code jmx uses to listen detected wrong address.
To fix this add the following to cassandra nodes startup script instances:
-Djava.rmi.server.hostname=127.0.0.1
(change 127.0.0.1 to actual internal address of cassandra node)

Re: Dual NIC server problems

Posted by aaron morton <aa...@thelastpickle.com>.
The code for nodetool appears to just pass the host value through to the NodeProbe. Was there anything else in the stack trace ? 

If you use the host name of the machine rather than ip what happens?

cassandra-env.sh includes a link to this page about getting JMX running with firewalls
http://blogs.sun.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole 
Perhaps the info on  java.rmi.server.hostname may be helpful 

Hope that helps.
Aaron


On 14 Dec 2010, at 21:36, Arjan van Ham wrote:

> 
> I have a cluster with three nodes, version 0.7.0 RC2. Each node has dual NIC's, eth0 to the internet and eth1 to a private network (192.168.1.xxx). The outside NIC  on each node is frewalled using iptables, only port 22 is allowed through. My cassandra.yaml configuration file refers only to the internal hostnames that resolve to 192.168.1.xxx addresses.
> 
> Most of my Cassandra installation works fine, except for a few nodetool commands. It seems that they are trying to access the servers over the outside (firewalled) NIC and thus fail. If I disable the firewall they work fine. 
> 
> For example:
> 
> [cassandra@node11 ~]$ nodetool -h 192.168.1.153 decommission
> Error connecting to remote JMX agent!
> java.rmi.ConnectException: Connection refused to host: 72.52.199.70; nested exception is: 
>        java.net.ConnectException: Connection timed out
>        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
>        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
>        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
>        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
>        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
>        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
>        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
>        at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:115)
>        at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:91)
> 
> 
> And if I disable the firewall on the remote node it works right away.
> 
> 
> Any ideas anyone?
> 
> Thanks.