You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Robert Hou (JIRA)" <ji...@apache.org> on 2017/10/03 22:00:04 UTC
[jira] [Closed] (DRILL-5813) A query that includes sort loses Drill
connection. Drill sometimes crashes.
[ https://issues.apache.org/jira/browse/DRILL-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Hou closed DRILL-5813.
-----------------------------
I have verified this.
> A query that includes sort loses Drill connection. Drill sometimes crashes.
> ----------------------------------------------------------------------------
>
> Key: DRILL-5813
> URL: https://issues.apache.org/jira/browse/DRILL-5813
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.11.0
> Reporter: Robert Hou
> Assignee: Vlad Rozov
> Attachments: drillbit.log, drill.log
>
>
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.enable_decimal_data_type` = true;
> select count(*) from (select * from dfs.`/drill/testdata/resource-manager/all_types_large` order by missing11) d where d.missing3 is false;
> {noformat}
> This query has passed before when the number of threads and amount of memory is restricted. With more threads and memory, the query does not complete execution.
> Here is the stack trace:
> {noformat}
> Exception occurred with closed channel. Connection: /10.10.100.190:59281 <--> /10.10.100.190:31010 (user client)
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
> at oadd.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
> at oadd.io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407)
> at oadd.io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:32)
> at oadd.io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:792)
> at oadd.io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280)
> at oadd.io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26)
> at oadd.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
> at oadd.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> User Error Occurred: Connection /10.10.100.190:59281 <--> /10.10.100.190:31010 (user client) closed unexpectedly. Drillbit down?
> oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection /10.10.100.190:59281 <--> /10.10.100.190:31010 (user client) closed un
> expectedly. Drillbit down?
> [Error Id: b97704a4-b8f0-4cd0-b428-2cf1bcf39a1d ]
> at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> [#14] Query failed:
> oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection /10.10.100.190:59281 <--> /10.10.100.190:31010 (user client) closed un
> expectedly. Drillbit down?
> [Error Id: b97704a4-b8f0-4cd0-b428-2cf1bcf39a1d ]
> at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> SYSTEM ERROR: IllegalArgumentException: Attempted to send a message when connection is no longer valid.
> Query submission to Drillbit failed.
> [Error Id: 72a44d5e-0e4a-4343-ac78-ff4bb1cb1219 ]
> oadd.org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalArgumentException: Attempted to send a message when connection is no longer va
> lid.
> Query submission to Drillbit failed.
> [Error Id: 72a44d5e-0e4a-4343-ac78-ff4bb1cb1219 ]
> at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$SubmissionListener.failed(QueryResultHandler.java:294)
> at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:124)
> at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:91)
> at oadd.org.apache.drill.exec.rpc.BasicClient.send(BasicClient.java:174)
> at oadd.org.apache.drill.exec.rpc.user.UserClient.submitQuery(UserClient.java:129)
> at oadd.org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:794)
> at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:543)
> at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at oadd.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at oadd.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> at oadd.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at oadd.org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> at org.apache.drill.test.framework.DrillTestJdbc.executeSetupQuery(DrillTestJdbc.java:181)
> at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: oadd.org.apache.drill.exec.rpc.RpcException: Failure sending message.
> ... 22 more
> Caused by: java.lang.IllegalArgumentException: Attempted to send a message when connection is no longer valid.
> at oadd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at oadd.org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:88)
> at oadd.org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:162)
> at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:117)
> ... 21 more
> [#15] Query failed:
> oadd.org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalArgumentException: Attempted to send a message when connection is no longer va
> lid.
> Query submission to Drillbit failed.
> [Error Id: 72a44d5e-0e4a-4343-ac78-ff4bb1cb1219 ]
> at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$SubmissionListener.failed(QueryResultHandler.java:294)
> at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:124)
> at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:91)
> at oadd.org.apache.drill.exec.rpc.BasicClient.send(BasicClient.java:174)
> at oadd.org.apache.drill.exec.rpc.user.UserClient.submitQuery(UserClient.java:129)
> at oadd.org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:794)
> at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:543)
> at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> at oadd.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> at oadd.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> at org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> at oadd.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> at oadd.org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> at org.apache.drill.test.framework.DrillTestJdbc.executeSetupQuery(DrillTestJdbc.java:181)
> at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: oadd.org.apache.drill.exec.rpc.RpcException: Failure sending message.
> ... 22 more
> Caused by: java.lang.IllegalArgumentException: Attempted to send a message when connection is no longer valid.
> at oadd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at oadd.org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:88)
> at oadd.org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:162)
> at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:117)
> ... 21 more
> Failed while running cleanup query. Not returning connection to pool.
> org.apache.drill.jdbc.AlreadyClosedSqlException: ResultSet is already closed.
> at org.apache.drill.jdbc.impl.DrillResultSetImpl.throwIfClosed(DrillResultSetImpl.java:96)
> at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:128)
> at org.apache.drill.test.framework.DrillTestJdbc.executeSetupQuery(DrillTestJdbc.java:185)
> at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The first error in drillbit.log:
> {noformat}
> 2017-09-22 13:18:51,720 [BitServer-3] WARN o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type 1 took longer than 500ms. Actual duration was 197783ms.
> 2017-09-22 13:25:51,944 [BitServer-5] WARN o.a.drill.exec.rpc.data.DataServer - Message of mode PING of rpc type 0 took longer than 500ms. Actual duration was 394057ms.
> 2017-09-22 13:38:40,957 [BitServer-5] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.100.190:31012 <--> /10.10.100.190:43721 (data server). Closing connection.
> java.lang.OutOfMemoryError: Java heap space
> 2017-09-22 13:38:53,698 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.100.190:31010 <--> /10.10.100.190:59281 (user server). Closing connection.
> java.lang.OutOfMemoryError: Java heap space
> 2017-09-22 13:42:14,255 [UserServer-1] DEBUG o.a.drill.exec.work.foreman.Foreman - 263aa12c-a069-ef76-02ff-6314ff72dca7: State change requested RUNNING --> CANCELLATION_REQUESTED
> 2017-09-22 13:46:43,615 [263aa12c-a069-ef76-02ff-6314ff72dca7:frag:2:1] ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. Information message: Unable to handle out of memory condition in FragmentExecutor.
> java.lang.OutOfMemoryError: Java heap space
> 2017-09-22 13:52:25,011 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2017-09-22 13:52:43,818 [Drillbit-ShutdownHook#0] DEBUG o.apache.drill.exec.server.Drillbit - Shutdown begun.
> 2017-09-22 13:53:00,684 [BitClient-1] ERROR o.apache.drill.exec.rpc.BasicClient - Unable to maintain connection /10.10.100.190:43721 <--> qa-node190.qa.lab/10.10.100.190:31012 (data client). Closing connection.
> {noformat}
> sing cln q47
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)