You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Achuth Raghavendra <ac...@microsoft.com.INVALID> on 2017/07/16 03:22:26 UTC
Unexpected exception from downstream while disposing evaluator.
I see exceptions while disposing a evaluator when a Task is complete. is this expected ?.
/// <summary>
/// Called when [next].
/// </summary>
/// <param name="value">The value.</param>
public void OnNext(ICompletedTask value)
{
....
value.ActiveContext.Dispose(); // this will dispose the evaluator.
}
INFO: Unexpected exception from downstream. channel: [id: 0x6075988d, /100.96.37.97:56404 => /100.96.14.37:2014] local: /100.96.14.37:2014 remote: /100.96.37.97:56404
Jul 15, 2017 8:11:34 PM org.apache.reef.wake.remote.transport.netty.NettyChannelHandler exceptionCaught
WARNING: Unexpected exception from downstream.
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:446)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:879)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)
RE: Unexpected exception from downstream while disposing evaluator.
Posted by Achuth Raghavendra <ac...@microsoft.com.INVALID>.
Full logs:
Jul 15, 2017 9:01:34 PM org.apache.reef.javabridge.generic.JobDriver$CompletedTaskHandler onNext
INFO: CLR CompletedTaskHandler handler set, handling things with CLR handler.
Jul 15, 2017 9:01:34 PM org.apache.reef.javabridge.EvaluatorRequestorBridge submit
INFO: submitting evaluator request org.apache.reef.driver.evaluator.EvaluatorRequest@609d3f69
Jul 15, 2017 9:01:34 PM org.apache.hadoop.yarn.util.RackResolver coreResolve
INFO: Resolved bn2phy040411345 to /BN2/rack-04041-13
Jul 15, 2017 9:01:34 PM org.apache.reef.javabridge.generic.JobDriver$CompletedEvaluatorHandler onNext
INFO: Completed Evaluator container_e3691_1499961928063_0044_01_000106
Jul 15, 2017 9:01:34 PM org.apache.reef.javabridge.generic.JobDriver$CompletedEvaluatorHandler onNext
INFO: Handling the event of completed evaluator in CLR bridge.
Jul 15, 2017 9:01:34 PM org.apache.reef.wake.remote.transport.netty.NettyChannelHandler exceptionCaught
INFO: Unexpected exception from downstream. channel: [id: 0x53195a7d, /100.96.37.97:31476 => /100.96.14.37:2014] local: /100.96.14.37:2014 remote: /100.96.37.97:31476
Jul 15, 2017 9:01:34 PM org.apache.reef.wake.remote.transport.netty.NettyChannelHandler exceptionCaught
WARNING: Unexpected exception from downstream.
java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:446)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:879)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)
Jul 15, 2017 9:01:34 PM org.apache.reef.wake.remote.transport.netty.AbstractNettyEventListener exceptionCaught
WARNING: ExceptionEvent: local: /100.96.14.37:2014 remote: /100.96.37.97:31476 :: java.io.IOException: An existing connection was forcibly closed by the remote host
Jul 15, 2017 9:01:36 PM org.apache.reef.runtime.yarn.driver.DefaultRackNameFormatter getRackName
-----Original Message-----
From: Achuth Raghavendra [mailto:acragh@microsoft.com.INVALID]
Sent: Monday, July 17, 2017 8:08 AM
To: dev@reef.apache.org
Subject: RE: Unexpected exception from downstream while disposing evaluator.
In the java bridge component logs.
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Monday, July 17, 2017 7:54 AM
To: REEF Developers Mailinglist <de...@reef.apache.org>>
Subject: Re: Unexpected exception from downstream while disposing evaluator.
Where is this exception thrown? Driver or Evaluator?
Markus
On Sat, Jul 15, 2017 at 8:22 PM, Achuth Raghavendra <ac...@microsoft.com.invalid>> wrote:
> I see exceptions while disposing a evaluator when a Task is complete. is this expected ?.
>
> /// <summary>
> /// Called when [next].
> /// </summary>
> /// <param name="value">The value.</param>
> public void OnNext(ICompletedTask value)
> {
> ....
> value.ActiveContext.Dispose(); // this will dispose the evaluator.
> }
>
>
> INFO: Unexpected exception from downstream. channel: [id: 0x6075988d,
> /100.96.37.97:56404 => /100.96.14.37:2014] local: /100.96.14.37:2014
> remote: /100.96.37.97:56404 Jul 15, 2017 8:11:34 PM
> org.apache.reef.wake.remote.transport.netty.NettyChannelHandler
> exceptionCaught
> WARNING: Unexpected exception from downstream.
> java.io.IOException: An existing connection was forcibly closed by the remote host
> at sun.nio.ch.SocketDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:446)
> at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:879)
> at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
RE: Unexpected exception from downstream while disposing evaluator.
Posted by Achuth Raghavendra <ac...@microsoft.com.INVALID>.
In the java bridge component logs.
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Monday, July 17, 2017 7:54 AM
To: REEF Developers Mailinglist <de...@reef.apache.org>
Subject: Re: Unexpected exception from downstream while disposing evaluator.
Where is this exception thrown? Driver or Evaluator?
Markus
On Sat, Jul 15, 2017 at 8:22 PM, Achuth Raghavendra <ac...@microsoft.com.invalid> wrote:
> I see exceptions while disposing a evaluator when a Task is complete. is this expected ?.
>
> /// <summary>
> /// Called when [next].
> /// </summary>
> /// <param name="value">The value.</param>
> public void OnNext(ICompletedTask value)
> {
> ....
> value.ActiveContext.Dispose(); // this will dispose the evaluator.
> }
>
>
> INFO: Unexpected exception from downstream. channel: [id: 0x6075988d,
> /100.96.37.97:56404 => /100.96.14.37:2014] local: /100.96.14.37:2014
> remote: /100.96.37.97:56404 Jul 15, 2017 8:11:34 PM
> org.apache.reef.wake.remote.transport.netty.NettyChannelHandler
> exceptionCaught
> WARNING: Unexpected exception from downstream.
> java.io.IOException: An existing connection was forcibly closed by the remote host
> at sun.nio.ch.SocketDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:446)
> at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:879)
> at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
Re: Unexpected exception from downstream while disposing evaluator.
Posted by Markus Weimer <ma...@weimo.de>.
Where is this exception thrown? Driver or Evaluator?
Markus
On Sat, Jul 15, 2017 at 8:22 PM, Achuth Raghavendra
<ac...@microsoft.com.invalid> wrote:
> I see exceptions while disposing a evaluator when a Task is complete. is this expected ?.
>
> /// <summary>
> /// Called when [next].
> /// </summary>
> /// <param name="value">The value.</param>
> public void OnNext(ICompletedTask value)
> {
> ....
> value.ActiveContext.Dispose(); // this will dispose the evaluator.
> }
>
>
> INFO: Unexpected exception from downstream. channel: [id: 0x6075988d, /100.96.37.97:56404 => /100.96.14.37:2014] local: /100.96.14.37:2014 remote: /100.96.37.97:56404
> Jul 15, 2017 8:11:34 PM org.apache.reef.wake.remote.transport.netty.NettyChannelHandler exceptionCaught
> WARNING: Unexpected exception from downstream.
> java.io.IOException: An existing connection was forcibly closed by the remote host
> at sun.nio.ch.SocketDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:446)
> at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:879)
> at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)