You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Sean Hsuan-Yi Chu (JIRA)" <ji...@apache.org> on 2015/05/15 07:22:59 UTC
[jira] [Resolved] (DRILL-3085) In ExternalSortBatch, Memory Leak in
Runtime Generation Code
[ https://issues.apache.org/jira/browse/DRILL-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Hsuan-Yi Chu resolved DRILL-3085.
--------------------------------------
Resolution: Fixed
> In ExternalSortBatch, Memory Leak in Runtime Generation Code
> ------------------------------------------------------------
>
> Key: DRILL-3085
> URL: https://issues.apache.org/jira/browse/DRILL-3085
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
> Reporter: Sean Hsuan-Yi Chu
> Assignee: Sean Hsuan-Yi Chu
> Priority: Critical
>
> This case is related to DRILL-3065.
> In ExternalSortBatch, we have an MSorter to do sorting thing. In this class, there are two SelectionVector4, vector4 and aux. If we fail at the time just after either gets new memory, the close() method would fail to clean their allocated memory properly.
> To reproduce this problem, inject an exception at the last step of MSortTemplate.setup()
> Detailed Information:
> 1. Query:
> select n_name from cp.`tpch/nation.parquet` order by n_name
> *. Using this query alone cannot help reproduce the issue. We still need to inject the exception at the right place.
> 2. Data:
> cp.`tpch/nation.parquet`
> 3. Log:
> java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are
> zone init allocated delta
> private 0 0 0
> shared 3221225472 3195686243 25539229.
> at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)
> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:386)
> at org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:175)
> at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)
> at com.google.common.io.Closeables.close(Closeables.java:77)
> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:292)
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:238)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
> at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Exception in thread "Drillbit-ShutdownHook#0" java.lang.RuntimeException: Caught exception closing Drillbit started from
> org.apache.drill.common.StackTrace.<init>:36
> org.apache.drill.exec.server.Drillbit.run:250
> org.apache.drill.BaseTestQuery.openClient:180
> org.apache.drill.BaseTestQuery.setupDefaultTestCluster:116
> sun.reflect.NativeMethodAccessorImpl.invoke0:-2
> sun.reflect.NativeMethodAccessorImpl.invoke:57
> sun.reflect.DelegatingMethodAccessorImpl.invoke:43
> java.lang.reflect.Method.invoke:606
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall:47
> org.junit.internal.runners.model.ReflectiveCallable.run:12
> org.junit.runners.model.FrameworkMethod.invokeExplosively:44
> org.junit.internal.runners.statements.RunBefores.evaluate:24
> org.junit.internal.runners.statements.RunAfters.evaluate:27
> org.junit.runners.ParentRunner.run:309
> org.junit.runner.JUnitCore.run:160
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs:74
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart:211
> com.intellij.rt.execution.junit.JUnitStarter.main:67
> sun.reflect.NativeMethodAccessorImpl.invoke0:-2
> sun.reflect.NativeMethodAccessorImpl.invoke:57
> sun.reflect.DelegatingMethodAccessorImpl.invoke:43
> java.lang.reflect.Method.invoke:606
> at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:333)
> Caused by: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are
> zone init allocated delta
> private 0 0 0
> shared 3221225472 3195686243 25539229.
> at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200)
> at org.apache.drill.exec.memory.Accountor.close(Accountor.java:386)
> at org.apache.drill.exec.memory.TopLevelAllocator.close(TopLevelAllocator.java:175)
> at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:75)
> at com.google.common.io.Closeables.close(Closeables.java:77)
> at com.google.common.io.Closeables.closeQuietly(Closeables.java:108)
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:292)
> at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:330)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)