You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Abhishek Ravi (JIRA)" <ji...@apache.org> on 2018/11/11 23:56:00 UTC
[jira] [Comment Edited] (DRILL-6838) Query with Runtime Filter
fails with IllegalStateException: Memory was leaked by query
[ https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683072#comment-16683072 ]
Abhishek Ravi edited comment on DRILL-6838 at 11/11/18 11:55 PM:
-----------------------------------------------------------------
[~weijie] - thanks for the changes but I am afraid the issue still exists even after the latest changes.
h3. Execution Failures:
{noformat}
Query: /root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
... 1 more
{noformat}
{noformat}
Query: /root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_02.q
select
supp_nation,
cust_nation,
l_year,
sum(volume) as revenue
from
(
select
n1.n_name as supp_nation,
n2.n_name as cust_nation,
extract(year from l.l_shipdate) as l_year,
l.l_extendedprice * (1 - l.l_discount) as volume
from
supplier s,
lineitem l,
orders o,
customer c,
nation n1,
nation n2
where
s.s_suppkey = l.l_suppkey
and o.o_orderkey = l.l_orderkey
and c.c_custkey = o.o_custkey
and s.s_nationkey = n1.n_nationkey
and c.c_nationkey = n2.n_nationkey
and (
(n1.n_name = 'EGYPT' and n2.n_name = 'UNITED STATES')
or (n1.n_name = 'UNITED STATES' and n2.n_name = 'EGYPT')
)
and l.l_shipdate between date '1995-01-01' and date '1996-12-31'
) as shipping
group by
supp_nation,
cust_nation,
l_year
order by
supp_nation,
cust_nation,
l_year
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
Fragment 5:0
Please, refer to logs for more information.
[Error Id: 98277144-fdb3-45df-ae5f-6578e50d823a on qa-node188.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
Fragment 5:0
Please, refer to logs for more information.
[Error Id: 98277144-fdb3-45df-ae5f-6578e50d823a on qa-node188.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
... 1 more
{noformat}
was (Author: aravi5):
[~weijie] - thanks for the changes but I am afraid the issue still exists even after the latest changes.
h1. h3. Execution Failures:
{noformat}
Query: /root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (4194304)
Allocator(frag:3:41) 8000000/4194304/81947456/60421075224 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
... 1 more
{noformat}
{noformat}
Query: /root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_02.q
select
supp_nation,
cust_nation,
l_year,
sum(volume) as revenue
from
(
select
n1.n_name as supp_nation,
n2.n_name as cust_nation,
extract(year from l.l_shipdate) as l_year,
l.l_extendedprice * (1 - l.l_discount) as volume
from
supplier s,
lineitem l,
orders o,
customer c,
nation n1,
nation n2
where
s.s_suppkey = l.l_suppkey
and o.o_orderkey = l.l_orderkey
and c.c_custkey = o.o_custkey
and s.s_nationkey = n1.n_nationkey
and c.c_nationkey = n2.n_nationkey
and (
(n1.n_name = 'EGYPT' and n2.n_name = 'UNITED STATES')
or (n1.n_name = 'UNITED STATES' and n2.n_name = 'EGYPT')
)
and l.l_shipdate between date '1995-01-01' and date '1996-12-31'
) as shipping
group by
supp_nation,
cust_nation,
l_year
order by
supp_nation,
cust_nation,
l_year
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
Fragment 5:0
Please, refer to logs for more information.
[Error Id: 98277144-fdb3-45df-ae5f-6578e50d823a on qa-node188.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
Fragment 5:0
Please, refer to logs for more information.
[Error Id: 98277144-fdb3-45df-ae5f-6578e50d823a on qa-node188.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (2097152)
Allocator(frag:5:0) 4000000/2097152/57691136/30078951604 (res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
... 1 more
{noformat}
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
> --------------------------------------------------------------------------------------
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR -> [https://github.com/apache/drill/pull/1504]
> Reporter: Abhishek Ravi
> Assignee: weijie.tong
> Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(frag:5:38) 6000000/1048576/77833920/50062914560 (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010] (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can easily reproduce the issue). I have attached logs for a simpler sample query that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)