You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 박상길 <ha...@gmail.com> on 2011/06/22 11:02:26 UTC
Strange Connection error of nodetool
Hi.
I'm running 5 cassandra nodes. Say, the addresses are 112.234.123.111 ~ 112.234.123.115; the real address is different.
When I run nodetool, the one node of address 112.234.123.112 has failed to connect. Showing error message like this.
iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
Error connection to remote JMX agent!
java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
The host to connect address differ! I had tried to query 112.* but, the nodetool tried to connect 122.*. It happened just one machine. All other machines works fine.
And I can connect to 112.234.123.112 by cassandra-cli or other tools using other port (such as 22 of ssh, 80 of http). It has trouble only on nodetool.
Does anyone has an idea?
I'll paste the full stack trace below.
iPark:~ hayarobi$ nodetool --host 112.234.123.111 ring
Address Status State Load Owns Token
136112946768375000000000000000000000000
112.234.123.111 Up Normal 725.01 KB 20.00% 0
112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
iPark:~ hayarobi$ nodetool --host 112.234.123.115 ring
Address Status State Load Owns Token
136112946768375000000000000000000000000
112.234.123.111 Up Normal 725.01 KB 20.00% 0
112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
iPark:~ hayarobi$ nodetool --host 112.234.123.114 ring
Address Status State Load Owns Token
136112946768375000000000000000000000000
112.234.123.111 Up Normal 725.01 KB 20.00% 0
112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
Error connection to remote JMX agent!
java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 10 more
Re: Strange Connection error of nodetool
Posted by 박상길 <ha...@kthcorp.com>.
Thank you, Nick.
I found that the /etc/hosts file was wrong, after reading your answer.
I fixed the wrong host ip address, and it works now.
2011. 6. 23., 오전 1:46, Nick Bailey 작성:
> This is almost certainly caused by the weird connection process JMX
> uses. JMX actually uses a 2 connection process, the second connection
> is determined by the 'JVM_OPTS="$JVM_OPTS
> -Djava.rmi.server.hostname=<public name>"' setting in your
> cassandra-env.sh configuration file.
>
> By default that setting is commented out and the hostname will default
> to whatever the local machine thinks it's hostname is. You should be
> able to fix this by uncommenting that setting in cassandra-env.sh and
> using the correct hostname for that setting.
>
> On Wed, Jun 22, 2011 at 4:02 AM, 박상길 <ha...@gmail.com> wrote:
>> Hi.
>>
>> I'm running 5 cassandra nodes. Say, the addresses are 112.234.123.111 ~ 112.234.123.115; the real address is different.
>> When I run nodetool, the one node of address 112.234.123.112 has failed to connect. Showing error message like this.
>>
>> iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
>> Error connection to remote JMX agent!
>> java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
>>
>> The host to connect address differ! I had tried to query 112.* but, the nodetool tried to connect 122.*. It happened just one machine. All other machines works fine.
>> And I can connect to 112.234.123.112 by cassandra-cli or other tools using other port (such as 22 of ssh, 80 of http). It has trouble only on nodetool.
>>
>> Does anyone has an idea?
>>
>> I'll paste the full stack trace below.
>>
>> iPark:~ hayarobi$ nodetool --host 112.234.123.111 ring
>> Address Status State Load Owns Token
>> 136112946768375000000000000000000000000
>> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
>> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
>> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
>> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
>> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
>> iPark:~ hayarobi$ nodetool --host 112.234.123.115 ring
>> Address Status State Load Owns Token
>> 136112946768375000000000000000000000000
>> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
>> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
>> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
>> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
>> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
>> iPark:~ hayarobi$ nodetool --host 112.234.123.114 ring
>> Address Status State Load Owns Token
>> 136112946768375000000000000000000000000
>> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
>> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
>> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
>> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
>> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
>> iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
>> Error connection to remote JMX agent!
>> java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
>> java.net.ConnectException: Connection refused
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
>> at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
>> at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
>> at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
>> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:529)
>> at java.net.Socket.connect(Socket.java:478)
>> at java.net.Socket.<init>(Socket.java:375)
>> at java.net.Socket.<init>(Socket.java:189)
>> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
>> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
>> ... 10 more
>>
>>
Re: Strange Connection error of nodetool
Posted by Nick Bailey <ni...@datastax.com>.
This is almost certainly caused by the weird connection process JMX
uses. JMX actually uses a 2 connection process, the second connection
is determined by the 'JVM_OPTS="$JVM_OPTS
-Djava.rmi.server.hostname=<public name>"' setting in your
cassandra-env.sh configuration file.
By default that setting is commented out and the hostname will default
to whatever the local machine thinks it's hostname is. You should be
able to fix this by uncommenting that setting in cassandra-env.sh and
using the correct hostname for that setting.
On Wed, Jun 22, 2011 at 4:02 AM, 박상길 <ha...@gmail.com> wrote:
> Hi.
>
> I'm running 5 cassandra nodes. Say, the addresses are 112.234.123.111 ~ 112.234.123.115; the real address is different.
> When I run nodetool, the one node of address 112.234.123.112 has failed to connect. Showing error message like this.
>
> iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
> Error connection to remote JMX agent!
> java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
>
> The host to connect address differ! I had tried to query 112.* but, the nodetool tried to connect 122.*. It happened just one machine. All other machines works fine.
> And I can connect to 112.234.123.112 by cassandra-cli or other tools using other port (such as 22 of ssh, 80 of http). It has trouble only on nodetool.
>
> Does anyone has an idea?
>
> I'll paste the full stack trace below.
>
> iPark:~ hayarobi$ nodetool --host 112.234.123.111 ring
> Address Status State Load Owns Token
> 136112946768375000000000000000000000000
> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
> iPark:~ hayarobi$ nodetool --host 112.234.123.115 ring
> Address Status State Load Owns Token
> 136112946768375000000000000000000000000
> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
> iPark:~ hayarobi$ nodetool --host 112.234.123.114 ring
> Address Status State Load Owns Token
> 136112946768375000000000000000000000000
> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
> iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
> Error connection to remote JMX agent!
> java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
> at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
> at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
> at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:529)
> at java.net.Socket.connect(Socket.java:478)
> at java.net.Socket.<init>(Socket.java:375)
> at java.net.Socket.<init>(Socket.java:189)
> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> ... 10 more
>
>
Re: Strange Connection error of nodetool
Posted by aaron morton <aa...@thelastpickle.com>.
Check the list here
http://wiki.apache.org/cassandra/JmxGotchas
I *think* the jmx server tells the client to connect back on another host/port.
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 22 Jun 2011, at 21:02, 박상길 wrote:
> Hi.
>
> I'm running 5 cassandra nodes. Say, the addresses are 112.234.123.111 ~ 112.234.123.115; the real address is different.
> When I run nodetool, the one node of address 112.234.123.112 has failed to connect. Showing error message like this.
>
> iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
> Error connection to remote JMX agent!
> java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
>
> The host to connect address differ! I had tried to query 112.* but, the nodetool tried to connect 122.*. It happened just one machine. All other machines works fine.
> And I can connect to 112.234.123.112 by cassandra-cli or other tools using other port (such as 22 of ssh, 80 of http). It has trouble only on nodetool.
>
> Does anyone has an idea?
>
> I'll paste the full stack trace below.
>
> iPark:~ hayarobi$ nodetool --host 112.234.123.111 ring
> Address Status State Load Owns Token
> 136112946768375000000000000000000000000
> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
> iPark:~ hayarobi$ nodetool --host 112.234.123.115 ring
> Address Status State Load Owns Token
> 136112946768375000000000000000000000000
> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
> iPark:~ hayarobi$ nodetool --host 112.234.123.114 ring
> Address Status State Load Owns Token
> 136112946768375000000000000000000000000
> 112.234.123.111 Up Normal 725.01 KB 20.00% 0
> 112.234.123.112 Up Normal 725.93 KB 20.00% 34028236692093800000000000000000000000
> 112.234.123.113 Up Normal 728.2 KB 20.00% 68056473384187700000000000000000000000
> 112.234.123.114 Up Normal 713.1 KB 20.00% 102084710076282000000000000000000000000
> 112.234.123.115 Up Normal 722.67 KB 20.00% 136112946768375000000000000000000000000
> iPark:~ hayarobi$ nodetool --host 112.234.123.112 ring
> Error connection to remote JMX agent!
> java.rmi.ConnectException: Connection refused to host: 122.234.123.112; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
> at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
> at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
> at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:529)
> at java.net.Socket.connect(Socket.java:478)
> at java.net.Socket.<init>(Socket.java:375)
> at java.net.Socket.<init>(Socket.java:189)
> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> ... 10 more
>