You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Khurram Faraaz (JIRA)" <ji...@apache.org> on 2018/05/25 08:29:00 UTC
[jira] [Updated] (DRILL-6441) IllegalStateException:
Allocator[ROOT] closed with outstanding child allocators.
[ https://issues.apache.org/jira/browse/DRILL-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Khurram Faraaz updated DRILL-6441:
----------------------------------
Description:
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
Adding details from Tim's email here
{noformat}
This issue is not related to DRILL-5922, however it is yet another way we leak memory. Looking at the code when you view or submit a query via the rest api as an anonymous user, a session is created with an allocator. If you call drillbit.sh stop during this time, the Drillbit can shutdown while the anonymous session is active and while a child allocator is being used. In this case you will see the error you reported below.{noformat}
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0: State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
child allocators: 8
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) ~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
{noformat}
was:
This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
{noformat}
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0: State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
child allocators: 8
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
child allocators: 0
ledgers: 0
reservations: 0
ledgers: 0
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) ~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
{noformat}
> IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
> --------------------------------------------------------------------------------
>
> Key: DRILL-6441
> URL: https://issues.apache.org/jira/browse/DRILL-6441
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.14.0
> Reporter: Khurram Faraaz
> Priority: Major
>
> This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh stop, it happens when BaseAllocator tries to close.
> Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab
> Adding details from Tim's email here
> {noformat}
> This issue is not related to DRILL-5922, however it is yet another way we leak memory. Looking at the code when you view or submit a query via the rest api as an anonymous user, a session is created with an allocator. If you call drillbit.sh stop during this time, the Drillbit can shutdown while the anonymous session is active and while a child allocator is being used. In this case you will see the error you reported below.{noformat}
>
> {noformat}
>
> 2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0: State to report: FINISHED
> Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
> 2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request.
> 2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss count: 6, hit rate: 70%
> 2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR o.a.d.exec.server.BootStrapContext - Error while closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
> Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
> child allocators: 8
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 0
> reservations: 0
> ledgers: 0
> reservations: 0
> at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) ~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> 2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)