You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Khurram Faraaz (JIRA)" <ji...@apache.org> on 2017/06/03 07:39:04 UTC
[jira] [Created] (DRILL-5561) IllegalArgumentException:
Self-suppression not permitted, seen when foreman Drillbit is killed
Khurram Faraaz created DRILL-5561:
-------------------------------------
Summary: IllegalArgumentException: Self-suppression not permitted, seen when foreman Drillbit is killed
Key: DRILL-5561
URL: https://issues.apache.org/jira/browse/DRILL-5561
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.11.0
Environment: 3 node Centos cluster
Reporter: Khurram Faraaz
Priority: Minor
Drill 1.11.0 running on three nodes
kill foreman Drillbit, [root@centos-01 ~]# kill -9 15852; while a long running SQL query (TPCDS Query 11) is under execution
drillbit.log from centos-01.qa.lab (foreman Drillbit) has no Exceptions or Errors logged in it.
snippet of drillbit.log from centos-02.qa.lab ( non foreman Drillbit )
{noformat}
2017-05-30 08:21:32,834 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:12:0: State change requested RUNNING --> FAILED
2017-05-30 08:21:32,834 [BitServer-9] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.202:31012 <--> /10.10.100.201:50773 (data server)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_65]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_65]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_65]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_65]
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:30) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:770) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-05-30 08:21:32,848 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:16:0: State change requested RUNNING --> FAILED
2017-05-30 08:21:32,850 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:14:0: State change requested RUNNING --> FAILED
2017-05-30 08:21:32,851 [BitClient-1] WARN o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
at org.apache.drill.common.DeferredException.addException(DeferredException.java:88) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:423) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:214) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:96) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.ListeningCommand$DeferredRpcOutcome.failed(ListeningCommand.java:50) ~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:127) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:134) ~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:74) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:64) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692) [hppc-0.7.1.jar:na]
at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:58) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed /10.10.100.202:46969 <--> centos-01.qa.lab/10.10.100.201:31012.
at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:164) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
... 18 common frames omitted
2017-05-30 08:21:32,852 [BitClient-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:10:0: State change requested RUNNING --> FAILED
2017-05-30 08:21:32,853 [BitClient-1] WARN o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
at org.apache.drill.common.DeferredException.addException(DeferredException.java:88) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97) ~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:423) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:214) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:96) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.ListeningCommand$DeferredRpcOutcome.failed(ListeningCommand.java:50) ~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:127) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:134) ~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:74) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:64) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692) [hppc-0.7.1.jar:na]
at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:58) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
...
...
2017-05-30 08:22:14,210 [BitServer-3] ERROR o.apache.drill.exec.rpc.BasicClient - Failed to establish connection
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:47) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:225) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:212) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_65]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_65]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
... 6 common frames omitted
{noformat}
snippet of drillbit.log from centos-04.qa.lab (another non foreman Drillbit)
{noformat}
2017-05-30 08:21:19,892 [26d2d481-d949-1de8-9497-c1492f086501:frag:8:2] INFO o.a.d.e.w.f.FragmentStatusReporter - 26d2d481-d949-1de8-9497-c1492f086501:8:2: State to report: RUNNING
2017-05-30 08:21:35,723 [BitServer-10] ERROR o.apache.drill.exec.rpc.BasicClient - Failed to establish connection
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:47) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:225) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:212) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_65]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_65]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
... 6 common frames omitted
{noformat}
Adding details from [~paul-rogers] email about this problem
{noformat}
The message appears to come from Netty, it is not a Drill string.
I think what this is saying is that we did something wrong in Netty and Netty is reporting a problem that we’ve violated its API rules.
We should figure out what we did wrong and fix it so that we don’t use Netty wrong.
Since we killed a Drillbit, the message should be “Connection to a.b.c.d dropped” — something the user can understand.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)