You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sergey Dobrovolskiy <se...@openbridge.com> on 2014/03/13 15:22:47 UTC

java.net.NoRouteToHostException: No route to host using Netty

Hi everybody,

I've setted up a cluster 0.9.0.1 version.

dedicated nimbus, zookeeper, two supervisors.

Have to say that everything works just fine with one supervisor. But 
when I add another one, task from one machine can't communicate with 
task on another machine, and because of that, messages are getting timed 
out.

I'm running everything on AWS using CentOS instances. I'v opend all 
ports for supervisor security group. Stoped iptables. But still no luck.

when I do |netstat -a| I can see |ESTABLISHED| connections between 
servers. I can also successfully |ping| each other.

No need to say that zookeeper works fine, and I can run |echo ruok | nc 
<ip> 2181| and I'm getting result.

But when Im checking worker log, I see those errors:

|2014-03-13 13:53:28 STDIO [ERROR] Mar 13, 2014 1:53:28 PM org.jboss.netty.channel.DefaultChannelPipeline
WARNING: An exception was thrown by a user handler while handling an exception event ([id: 0x4bb68a13] EXCEPTION: java.net.NoRouteToHostException: No route to host)
java.lang.IllegalArgumentException: timeout value is negative
     at java.lang.Thread.sleep(Native Method)
     at backtype.storm.messaging.netty.Client.reconnect(Client.java:94)
     at backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118)
     at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
     at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
     at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
     at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
     at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
     at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:744)
2014-03-13 13:53:29 b.s.m.n.StormClientHandler [INFO] Connection failed:
java.net.NoRouteToHostException: No route to host
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_51]
     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_51]
     at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148) ~[netty-3.6.3.Final.jar:na]
     at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104) ~[netty-3.6.3.Final.jar:na]
     at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78) ~[netty-3.6.3.Final.jar:na]
     at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[netty-3.6.3.Final.jar:na]
     at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41) ~[netty-3.6.3.Final.jar:na]
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
     at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
|

Re: java.net.NoRouteToHostException: No route to host using Netty

Posted by Sergey Dobrovolskiy <se...@openbridge.com>.
Thank you for pointing this out.

I'm trying to use the updated version, but having trouble building a 
release.

1. I cloned git repo
2. Run mvn -f pom.xml package
3. Tried to run it, but it gives me an error:
The storm client can only be run from within a release. You appear to be 
trying to run the client from a checkout of Storm's source code.

You can download a Storm release at http://storm-project.net/downloads.html

Obviously the release from this site wasn't updated.
Do you know how can I apply newer changes?


On 03/13/2014 10:38 AM, James Xu wrote:
> This is a known issue, already merged: 
> https://github.com/apache/incubator-storm/pull/41.
>
> - James
> On 2014年3月13日, at 下午10:22, Sergey Dobrovolskiy 
> <sergey@openbridge.com <ma...@openbridge.com>> wrote:
>
>> Hi everybody,
>>
>> I've setted up a cluster 0.9.0.1 version.
>>
>> dedicated nimbus, zookeeper, two supervisors.
>>
>> Have to say that everything works just fine with one supervisor. But 
>> when I add another one, task from one machine can't communicate with 
>> task on another machine, and because of that, messages are getting 
>> timed out.
>>
>> I'm running everything on AWS using CentOS instances. I'v opend all 
>> ports for supervisor security group. Stoped iptables. But still no luck.
>>
>> when I do |netstat -a| I can see |ESTABLISHED| connections between 
>> servers. I can also successfully |ping| each other.
>>
>> No need to say that zookeeper works fine, and I can run |echo ruok | 
>> nc <ip> 2181| and I'm getting result.
>>
>> But when Im checking worker log, I see those errors:
>>
>> |2014-03-13 13:53:28 STDIO [ERROR] Mar 13, 2014 1:53:28 PM org.jboss.netty.channel.DefaultChannelPipeline
>> WARNING: An exception was thrown by a user handler while handling an exception event ([id: 0x4bb68a13] EXCEPTION: java.net.NoRouteToHostException: No route to host)
>> java.lang.IllegalArgumentException: timeout value is negative
>>      at java.lang.Thread.sleep(Native Method)
>>      at backtype.storm.messaging.netty.Client.reconnect(Client.java:94)
>>      at backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118)
>>      at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
>>      at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
>>      at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
>>      at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
>>      at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>      at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>>      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>      at java.lang.Thread.run(Thread.java:744)
>> 2014-03-13 13:53:29 b.s.m.n.StormClientHandler [INFO] Connection failed:
>> java.net.NoRouteToHostException: No route to host
>>      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_51]
>>      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_51]
>>      at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148) ~[netty-3.6.3.Final.jar:na]
>>      at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104) ~[netty-3.6.3.Final.jar:na]
>>      at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78) ~[netty-3.6.3.Final.jar:na]
>>      at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[netty-3.6.3.Final.jar:na]
>>      at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41) ~[netty-3.6.3.Final.jar:na]
>>      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
>>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
>>      at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
>> |
>

Re: java.net.NoRouteToHostException: No route to host using Netty

Posted by James Xu <xu...@gmail.com>.
This is a known issue, already merged: https://github.com/apache/incubator-storm/pull/41.

- James
On 2014年3月13日, at 下午10:22, Sergey Dobrovolskiy <se...@openbridge.com> wrote:

> Hi everybody,
> 
> I've setted up a cluster 0.9.0.1 version.
> 
> dedicated nimbus, zookeeper, two supervisors.
> 
> Have to say that everything works just fine with one supervisor. But when I add another one, task from one machine can't communicate with task on another machine, and because of that, messages are getting timed out.
> 
> I'm running everything on AWS using CentOS instances. I'v opend all ports for supervisor security group. Stoped iptables. But still no luck.
> 
> when I do netstat -a I can see ESTABLISHED connections between servers. I can also successfully ping each other.
> 
> No need to say that zookeeper works fine, and I can run echo ruok | nc <ip> 2181 and I'm getting result.
> 
> But when Im checking worker log, I see those errors:
> 
> 2014-03-13 13:53:28 STDIO [ERROR] Mar 13, 2014 1:53:28 PM org.jboss.netty.channel.DefaultChannelPipeline
> WARNING: An exception was thrown by a user handler while handling an exception event ([id: 0x4bb68a13] EXCEPTION: java.net.NoRouteToHostException: No route to host)
> java.lang.IllegalArgumentException: timeout value is negative
>     at java.lang.Thread.sleep(Native Method)
>     at backtype.storm.messaging.netty.Client.reconnect(Client.java:94)
>     at backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118)
>     at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
>     at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
>     at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
>     at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
>     at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>     at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:744)
> 2014-03-13 13:53:29 b.s.m.n.StormClientHandler [INFO] Connection failed:
> java.net.NoRouteToHostException: No route to host
>     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_51]
>     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_51]
>     at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148) ~[netty-3.6.3.Final.jar:na]
>     at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104) ~[netty-3.6.3.Final.jar:na]
>     at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78) ~[netty-3.6.3.Final.jar:na]
>     at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) ~[netty-3.6.3.Final.jar:na]
>     at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41) ~[netty-3.6.3.Final.jar:na]
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
>     at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]