You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Kunal Khatua (JIRA)" <ji...@apache.org> on 2015/04/10 19:55:12 UTC
[jira] [Created] (DRILL-2750) Running 1 or more queries against
Drillbits having insufficient DirectMem renders the Drillbits in an
unusable state
Kunal Khatua created DRILL-2750:
-----------------------------------
Summary: Running 1 or more queries against Drillbits having insufficient DirectMem renders the Drillbits in an unusable state
Key: DRILL-2750
URL: https://issues.apache.org/jira/browse/DRILL-2750
Project: Apache Drill
Issue Type: Bug
Affects Versions: 0.9.0
Environment: RHEL 6.4
Reporter: Kunal Khatua
Priority: Critical
Fix For: 0.9.0
When running queries against a Drill cluster with limited DirectMem; if one or more queries fail due to insufficient memory, then even queries that should easily run within the allocated memory fail.
The initial failure when queries with large memory requirements fail:
2015-04-10 09:57:55 [pip0] ERROR PipSQuawkling fetchRows - [ 1 / 16_par1000 ] Failure while executing query.
java.sql.SQLException: Failure while executing query.
at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
at org.apache.drill.jdbc.DrillResultSet.next(DrillResultSet.java:85)
at PipSQuawkling.fetchRows(PipSQuawkling.java:319)
at PipSQuawkling.executeTest(PipSQuawkling.java:154)
at PipSQuawkling.run(PipSQuawkling.java:76)
Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment.[ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ]
[ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ]
at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:744)
After that, subsequent queries that should run, fail with the following:
2015-04-10 09:59:29 [pip0] ERROR PipSQuawkling executeQuery - [ 2 / rerun_06_par1000 ] exception while executing query: Failure while executing query.
java.sql.SQLException: exception while executing query: Failure while executing query.
at net.hydromatic.avatica.Helper.createException(Helper.java:40)
at net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:406)
at net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
at net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78)
at PipSQuawkling.executeQuery(PipSQuawkling.java:284)
at PipSQuawkling.executeTest(PipSQuawkling.java:144)
at PipSQuawkling.run(PipSQuawkling.java:76)
Caused by: java.sql.SQLException: Failure while executing query.
at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:105)
at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:44)
at net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
... 5 more
Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while trying to start remote fragment, You attempted to create a new child allocator with initial reservation 6000000 but only 110395 bytes of memory were available. [ 689006cb-d703-42c3-860d
-bfecc0a66312 on ucs-node10.perf.lab:31010 ]
at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)