You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Chris Westin (JIRA)" <ji...@apache.org> on 2015/04/10 20:37:12 UTC
[jira] [Assigned] (DRILL-2750) Running 1 or more queries against
Drillbits having insufficient DirectMem renders the Drillbits in an
unusable state
[ https://issues.apache.org/jira/browse/DRILL-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Westin reassigned DRILL-2750:
-----------------------------------
Assignee: Chris Westin
> Running 1 or more queries against Drillbits having insufficient DirectMem renders the Drillbits in an unusable state
> --------------------------------------------------------------------------------------------------------------------
>
> Key: DRILL-2750
> URL: https://issues.apache.org/jira/browse/DRILL-2750
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 0.9.0
> Environment: RHEL 6.4
> Reporter: Kunal Khatua
> Assignee: Chris Westin
> Priority: Critical
> Fix For: 0.9.0
>
>
> When running queries against a Drill cluster with limited DirectMem; if one or more queries fail due to insufficient memory, then even queries that should easily run within the allocated memory fail.
> The initial failure when queries with large memory requirements fail:
> 2015-04-10 09:57:55 [pip0] ERROR PipSQuawkling fetchRows - [ 1 / 16_par1000 ] Failure while executing query.
> java.sql.SQLException: Failure while executing query.
> at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
> at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
> at org.apache.drill.jdbc.DrillResultSet.next(DrillResultSet.java:85)
> at PipSQuawkling.fetchRows(PipSQuawkling.java:319)
> at PipSQuawkling.executeTest(PipSQuawkling.java:154)
> at PipSQuawkling.run(PipSQuawkling.java:76)
> Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment.[ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ]
> [ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ]
> at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
> at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
> at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
> at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
> at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:744)
> After that, subsequent queries that should run, fail with the following:
> 2015-04-10 09:59:29 [pip0] ERROR PipSQuawkling executeQuery - [ 2 / rerun_06_par1000 ] exception while executing query: Failure while executing query.
> java.sql.SQLException: exception while executing query: Failure while executing query.
> at net.hydromatic.avatica.Helper.createException(Helper.java:40)
> at net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:406)
> at net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
> at net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78)
> at PipSQuawkling.executeQuery(PipSQuawkling.java:284)
> at PipSQuawkling.executeTest(PipSQuawkling.java:144)
> at PipSQuawkling.run(PipSQuawkling.java:76)
> Caused by: java.sql.SQLException: Failure while executing query.
> at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:105)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:44)
> at net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
> ... 5 more
> Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while trying to start remote fragment, You attempted to create a new child allocator with initial reservation 6000000 but only 110395 bytes of memory were available. [ 689006cb-d703-42c3-860d
> -bfecc0a66312 on ucs-node10.perf.lab:31010 ]
> at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
> at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
> at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
> at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
> at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)