You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Jacques Nadeau (JIRA)" <ji...@apache.org> on 2016/04/19 22:31:25 UTC
[jira] [Resolved] (DRILL-4113) memory leak reported while handling
query or shutting down
[ https://issues.apache.org/jira/browse/DRILL-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacques Nadeau resolved DRILL-4113.
-----------------------------------
Resolution: Cannot Reproduce
> memory leak reported while handling query or shutting down
> ----------------------------------------------------------
>
> Key: DRILL-4113
> URL: https://issues.apache.org/jira/browse/DRILL-4113
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0
> Reporter: Chun Chang
> Priority: Critical
>
> With impersonation enabled, I've seen two memory leaks. One reported at query time, one at shutdown.
> At query time:
> {noformat}
> 2015-11-17 19:11:03,595 [29b413b7-958e-c1f3-9d37-c34f96e7bf6a:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 29b413b7-958e-c1f3-9d37-c34f96e7bf6a: use `dfs.window_functions`
> 2015-11-17 19:11:03,666 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
> 2015-11-17 19:11:03,666 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State to report: RUNNING
> 2015-11-17 19:11:03,669 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested RUNNING --> FAILED
> 2015-11-17 19:11:03,669 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested FAILED --> FAILED
> 2015-11-17 19:11:03,669 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 29b413b7-edbc-9722-120d-66ab3611f250:0:0: State change requested FAILED --> FINISHED
> 2015-11-17 19:11:03,674 [29b413b7-edbc-9722-120d-66ab3611f250:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are
> zone init allocated delta
> private 1000000 738112 261888
> shared 9999000000 9999261888 -261888.
> Fragment 0:0
> [Error Id: 6df67be9-69d4-4a3b-9eae-43ab2404c6d3 on drillats1.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are
> zone init allocated delta
> private 1000000 738112 261888
> shared 9999000000 9999261888 -261888.
> Fragment 0:0
> [Error Id: 6df67be9-69d4-4a3b-9eae-43ab2404c6d3 on drillats1.qa.lab:31010]
> at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534) ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are
> zone init allocated delta
> private 1000000 738112 261888
> shared 9999000000 9999261888 -261888.
> at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:199) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.memory.AccountorImpl.close(AccountorImpl.java:365) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:326) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:123) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:437) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:426) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:346) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:179) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> ... 5 common frames omitted
> 2015-11-17 19:11:03,723 [CONTROL-rpc-event-queue] WARN o.a.drill.exec.work.foreman.Foreman - Dropping request to move to COMPLETED state as query is already at FAILED state (which is terminal).
> 2015-11-17 19:11:03,724 [CONTROL-rpc-event-queue] WARN o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. 29b413b7-edbc-9722-120d-66ab3611f250:0:0 does not exist.
> {noformat}
> At shutdown:
> {noformat}
> Tue Nov 17 22:01:14 PST 2015 Terminating drillbit pid 32653
> 2015-11-17 22:01:14,569 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2015-11-17 22:01:15,559 [BitClient-2] INFO o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.100.171:56037 <--> /10.10.100.172:31012.
> 2015-11-17 22:01:15,559 [BitServer-6] INFO o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.100.171:47521 <--> /10.10.100.172:31011.
> 2015-11-17 22:01:15,574 [BitClient-4] INFO o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.100.171:59422 <--> /10.10.100.174:31012.
> 2015-11-17 22:01:15,575 [BitServer-2] INFO o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.100.171:55874 <--> /10.10.100.174:31011.
> 2015-11-17 22:01:15,582 [BitClient-3] INFO o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.100.171:49728 <--> /10.10.100.173:31012.
> 2015-11-17 22:01:15,582 [BitServer-5] INFO o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.100.171:43599 <--> /10.10.100.173:31011.
> 2015-11-17 22:01:15,603 [BitClient-1] INFO o.a.drill.exec.rpc.data.DataClient - Channel closed /10.10.100.171:56089 <--> /10.10.100.171:31012.
> 2015-11-17 22:01:15,603 [BitServer-3] INFO o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.100.171:52294 <--> /10.10.100.171:31011.
> 2015-11-17 22:01:16,610 [pool-1274-thread-1] INFO o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup io.netty.channel.epoll.EpollEventLoopGroup@44d08482 in 1005 ms
> 2015-11-17 22:01:16,610 [pool-1274-thread-2] INFO o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup io.netty.channel.epoll.EpollEventLoopGroup@1a10cc3c in 1007 ms
> 2015-11-17 22:01:16,610 [pool-1274-thread-1] INFO o.a.drill.exec.service.ServiceEngine - closed userServer in 1009 ms
> 2015-11-17 22:01:16,610 [pool-1274-thread-2] INFO o.a.drill.exec.service.ServiceEngine - closed dataPool in 1009 ms
> 2015-11-17 22:01:16,629 [Drillbit-ShutdownHook#0] WARN o.apache.drill.exec.server.Drillbit - Failure on close(): {}
> java.lang.RuntimeException: Exception while closing
> at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46) ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:127) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:44) ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:254) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:297) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> Caused by: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are
> zone init allocated delta
> private 0 0 0
> shared 34359738368 34358000256 1738112.
> at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:199) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.memory.AccountorImpl.close(AccountorImpl.java:365) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:179) ~[drill-memory-impl-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44) ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> ... 4 common frames omitted
> 2015-11-17 22:01:16,629 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (2059 ms).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)