[jira] [Created] (DRILL-3317) when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is just logged and nothing else is done

["Deneche A. Hakim (JIRA)" <ji...@apache.org>]

Deneche A. Hakim created DRILL-3317:
---------------------------------------

             Summary: when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is just logged and nothing else is done
                 Key: DRILL-3317
                 URL: https://issues.apache.org/jira/browse/DRILL-3317
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - RPC
            Reporter: Deneche A. Hakim
            Assignee: Jacques Nadeau
             Fix For: 1.2.0


Trying to reproduce DRILL-3241 I sometimes get the following error in the logs:
{noformat}
ERROR: Out of memory outside any particular fragment.
	at org.apache.drill.exec.rpc.data.DataResponseHandlerImpl.informOutOfMemory(DataResponseHandlerImpl.java:40)
	at org.apache.drill.exec.rpc.data.DataServer$2.handle(DataServer.java:227)
	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:87)
	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)

WARN: Failure allocating buffer on incoming stream due to memory limits.  Current Allocation: 1372678764.
	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:85)
	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
{noformat}

ProtobufLengthDecoder.decode() does call OutOfMemoryHandler.handle() which calls DataResponseHandlerImpl.informOutOfMemory() which just logs the error in the logs.

If we have fragments waiting for data they will be stuck waiting forever, and the query will hang (behavior observed in DRILL-3241



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)