You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by jinhong lu <lu...@gmail.com> on 2015/09/16 03:51:10 UTC

Betty connection fail


Hi, all

    I found a lot of netty fail in my storm cluster, Not just one topo, but all topo fail after this error occurs. 
    I use storm 0.9.4. please help.

here is the log :


2015-09-16T09:43:48.701+0800 b.s.m.n.Client [ERROR] connection attempt 60 to Netty-Client-gdc-storm03-storm./192.168.172.120:6727 failed: java.lang.RuntimeException: Returned channel was actually not established
2015-09-16T09:43:48.701+0800 b.s.m.n.Client [INFO] connection attempt 61 to Netty-Client-gdc-storm05-storm./192.168.172.122:6718 scheduled to run in 1174 ms
2015-09-16T09:43:48.701+0800 b.s.m.n.Client [INFO] connection attempt 61 to Netty-Client-gdc-storm03-storm./192.168.172.120:6727 scheduled to run in 1174 ms
2015-09-16T09:43:49.383+0800 b.s.m.n.StormClientErrorHandler [INFO] Connection failed Netty-Client-gdc-storm02-storm./192.168.172.119:6720
        at org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) [storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [storm-core-0.9.4.jar:0.9.4]
        at org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [storm-core-0.9.4.jar:0.9.4]
2015-09-16T09:43:49.534+0800 b.s.m.n.Client [ERROR] connection attempt 11 to Netty-Client-gdc-storm04-storm./192.168.172.121:6726 failed: java.lang.RuntimeException: Returned channel was actually not established


BR//lujinhong

Re: Betty connection fail

Posted by jinhong lu <lu...@gmail.com>.
I found another situation to cause netty reconnect:

 I remove many files, at the same time, I wc some files, and the files are very large(about 600G), so the IO is very very high(about 100M/s). No resource for netty to handle connection request, it leads reconnect.

So, I guess if load of the server is too high, it will fail netty. 


> 在 2015年9月16日,10:08,Erik Weathers <ew...@groupon.com> 写道:
> 
> 


Re: Betty connection fail

Posted by Erik Weathers <ew...@groupon.com>.
hi jinhong,

Please search through the archives for the last couple of days, looking for
"netty reconnect".  There have been a couple threads on this topic already.

- Erik

On Tue, Sep 15, 2015 at 6:51 PM, jinhong lu <lu...@gmail.com> wrote:

>
>
> Hi, all
>
>     I found a lot of netty fail in my storm cluster, Not just one topo,
> but all topo fail after this error occurs.
>     I use storm 0.9.4. please help.
>
> here is the log :
>
>
> 2015-09-16T09:43:48.701+0800 b.s.m.n.Client [ERROR] connection attempt 60
> to Netty-Client-gdc-storm03-storm./192.168.172.120:6727 failed:
> java.lang.RuntimeException: Returned channel was actually not established
> 2015-09-16T09:43:48.701+0800 b.s.m.n.Client [INFO] connection attempt 61
> to Netty-Client-gdc-storm05-storm./192.168.172.122:6718 scheduled to run
> in 1174 ms
> 2015-09-16T09:43:48.701+0800 b.s.m.n.Client [INFO] connection attempt 61
> to Netty-Client-gdc-storm03-storm./192.168.172.120:6727 scheduled to run
> in 1174 ms
> 2015-09-16T09:43:49.383+0800 b.s.m.n.StormClientErrorHandler [INFO]
> Connection failed Netty-Client-gdc-storm02-storm./192.168.172.119:6720
>         at
> org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
> [storm-core-0.9.4.jar:0.9.4]
>         at
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> [storm-core-0.9.4.jar:0.9.4]
>         at
> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> [storm-core-0.9.4.jar:0.9.4]
>         at
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> [storm-core-0.9.4.jar:0.9.4]
>         at
> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> [storm-core-0.9.4.jar:0.9.4]
>         at
> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> [storm-core-0.9.4.jar:0.9.4]
>         at
> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> [storm-core-0.9.4.jar:0.9.4]
> 2015-09-16T09:43:49.534+0800 b.s.m.n.Client [ERROR] connection attempt 11
> to Netty-Client-gdc-storm04-storm./192.168.172.121:6726 failed:
> java.lang.RuntimeException: Returned channel was actually not established
>
>
> BR//lujinhong