You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Yunmeng Ban <ba...@gmail.com> on 2014/08/11 15:57:02 UTC
Storm restart when data input rate is high and get connection fail
Hi everyone,
I met a problem while using netty as the communication tool. This exception
only occurs when input rate is high. (When input rate is low, everything
works fine.) Has anyone seen this exception? Please tell me the right way
to fix it. I've already add these into storm.yaml file
storm.messaging.transport: "backtype.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 419430400
storm.messaging.netty.server_worker_threads: 2
storm.messaging.netty.client_worker_threads: 2
storm.messaging.netty.buffer_size: 419430400
storm.messaging.netty.max_retries: 100
storm.messaging.netty.max_wait_ms: 20000
storm.messaging.netty.min_wait_ms: 10
topology.acker.executors: 20
But still get the error. I'm using storm-0.9.0-rc3
Any clue is appreciated.
2014-08-10 01:41:11 b.s.m.n.StormClientHandler [INFO] Connection failed:
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_13]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.7.0_13]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
~[na:1.7.0_13]
at sun.nio.ch.IOUtil.read(IOUtil.java:193) ~[na:1.7.0_13]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
~[na:1.7.0_13]
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
~[netty-3.6.3.Final.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_13]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_13]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
2014-08-10 01:41:11 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-08-10 01:41:11 b.s.m.n.StormClientHandler [INFO] failed to send
requests:
java.nio.channels.ClosedChannelException: null
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:381)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:349)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[netty-3.6.3.Final.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_13]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_13]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
2014-08-10 01:41:11 b.s.m.n.StormClientHandler [INFO] Connection failed:
java.nio.channels.ClosedChannelException: null
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:381)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:349)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
~[netty-3.6.3.Final.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_13]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_13]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
Thanks,
Yunmeng
Re: Storm restart when data input rate is high and get connection fail
Posted by Daria Mayorova <d....@gmail.com>.
Hi Yunmeng,
It's a pity you didn't receive any answers to your message. I am seeing the
same behaviour in my topology running on storm v0.9.3.
Did you manage to find the reason of the connection resets?
Any info is appreciated.
Thanks,
Daria
On Mon, Aug 11, 2014 at 3:57 PM, Yunmeng Ban <ba...@gmail.com> wrote:
> Hi everyone,
>
> I met a problem while using netty as the communication tool. This
> exception only occurs when input rate is high. (When input rate is low,
> everything works fine.) Has anyone seen this exception? Please tell me the
> right way to fix it. I've already add these into storm.yaml file
>
> storm.messaging.transport: "backtype.storm.messaging.netty.Context"
> storm.messaging.netty.buffer_size: 419430400
> storm.messaging.netty.server_worker_threads: 2
> storm.messaging.netty.client_worker_threads: 2
> storm.messaging.netty.buffer_size: 419430400
> storm.messaging.netty.max_retries: 100
> storm.messaging.netty.max_wait_ms: 20000
> storm.messaging.netty.min_wait_ms: 10
> topology.acker.executors: 20
>
> But still get the error. I'm using storm-0.9.0-rc3
> Any clue is appreciated.
>
> 2014-08-10 01:41:11 b.s.m.n.StormClientHandler [INFO] Connection failed:
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> ~[na:1.7.0_13]
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> ~[na:1.7.0_13]
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
> ~[na:1.7.0_13]
> at sun.nio.ch.IOUtil.read(IOUtil.java:193) ~[na:1.7.0_13]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
> ~[na:1.7.0_13]
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> ~[netty-3.6.3.Final.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_13]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_13]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
> 2014-08-10 01:41:11 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-08-10 01:41:11 b.s.m.n.StormClientHandler [INFO] failed to send
> requests:
> java.nio.channels.ClosedChannelException: null
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:381)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:349)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> [netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> [netty-3.6.3.Final.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_13]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_13]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
> 2014-08-10 01:41:11 b.s.m.n.StormClientHandler [INFO] Connection failed:
> java.nio.channels.ClosedChannelException: null
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:381)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:349)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> ~[netty-3.6.3.Final.jar:na]
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> ~[netty-3.6.3.Final.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_13]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_13]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
>
> Thanks,
> Yunmeng
>