[jira] [Created] (DRILL-6229) memory leak when output_batch_size is set to 4096

["Khurram Faraaz (JIRA)" <ji...@apache.org>]

Khurram Faraaz created DRILL-6229:
-------------------------------------

             Summary: memory leak when output_batch_size is set to 4096
                 Key: DRILL-6229
                 URL: https://issues.apache.org/jira/browse/DRILL-6229
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Flow
    Affects Versions: 1.12.0
            Reporter: Khurram Faraaz


When drill.exec.memory.operator.output_batch_size` = 4096 (4KB), we continuously see in each test run, the memory leak when tests are run from test framework.

Drill 1.13.0 git.commit.id=766315ea17377199897d685ab801edd38394fe01

alter system set `drill.exec.memory.operator.output_batch_size` = 4096;
To run flatten tests from drill test framework
bin/run_tests -s Functional/flatten_operators -g functional -d -t 500 -n 40

The below two tests time out and we see the memory leak
Timeouts:
/root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter24.q

Timeouts:
/root/drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional/flatten_operators/100000rows/filter19.q

oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection ... (user client) closed unexpectedly. Drillbit down?

Stack trace printed in drill test framework log
{noformat}
[Error Id: 54a16569-6555-4e92-aa5f-f76a8c11b31f ]
at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:376)
at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at oadd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1003)
at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:632)
at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:610)
at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:553)
at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1236)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
at oadd.io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at oadd.io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:602)
at oadd.io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:618)
at oadd.io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:35)
at oadd.io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:607)
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:748)
Channel is closed, discarding remaining 908 byte(s) in buffer.
Exception while closing client.
java.lang.RuntimeException: Exception while closing
at oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
at oadd.org.apache.drill.exec.client.DrillClient.close(DrillClient.java:459)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.closeOrWarn(DrillConnectionImpl.java:810)
at org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:823)
at org.apache.drill.jdbc.impl.DrillHandler.onConnectionClose(DrillHandler.java:36)
at oadd.org.apache.calcite.avatica.AvaticaConnection.close(AvaticaConnection.java:205)
at org.apache.drill.test.framework.ConnectionPool.close(ConnectionPool.java:102)
at org.apache.drill.test.framework.TestDriver.runTests(TestDriver.java:562)
at org.apache.drill.test.framework.TestDriver.main(TestDriver.java:100)
Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (4198400)
Allocator(ROOT) 0/4096/6713344/28631367680 (res/actual/peak/limit)

at oadd.org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:520)
at oadd.org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
... 8 more
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)