You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Deneche A. Hakim (JIRA)" <ji...@apache.org> on 2016/04/08 18:52:25 UTC
[jira] [Comment Edited] (DRILL-3842) JVM dies if drill attempts to
read too many files in the directory that blow up heap
[ https://issues.apache.org/jira/browse/DRILL-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228573#comment-15228573 ]
Deneche A. Hakim edited comment on DRILL-3842 at 4/8/16 4:51 PM:
-----------------------------------------------------------------
Although I was able to reproduce the issue on 1.2.0, it's no longer occurring in the latest master. It still takes more than 30mn to plan the query and the heap usage grows to 4GB on the Foreman node, but the query succeeds.
I suspect reading the parquet metadata for all 2M files is the cause, I will investigate if this is indeed the case
was (Author: adeneche):
Although I was able to reproduce the issue on 1.2.0, it's no longer occurring in the latest master. It still takes more than 30mn to plan the query and the heap usage grows to 4GB on the Foreman node, but the query succeeds.
I suspect reading the parquet metadata cache for all 2M files is the cause, I will investigate if this is indeed the case
> JVM dies if drill attempts to read too many files in the directory that blow up heap
> -------------------------------------------------------------------------------------
>
> Key: DRILL-3842
> URL: https://issues.apache.org/jira/browse/DRILL-3842
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Victoria Markman
> Assignee: Deneche A. Hakim
> Priority: Critical
>
> Run {{select count(*) from t1}} where t1 directory consists of 1.9 million little parquet files. The outcome: drillbit is dead and out of working set.
> 1. Client never got response back from the server
> 2. drillbit.log
> {code}
> 2015-09-25 17:56:55,935 [29fa756f-894d-0340-3661-b925bff0fe11:foreman] INFO o.a.d.exec.store.parquet.Metadata - Took 47999 ms to get file statuses
> 2015-09-25 18:43:19,871 [BitServer-4] INFO o.a.d.exec.rpc.control.ControlServer - RPC connection /10.10.88.135:31011 <--> /10.10.88.135:51675 (control server) timed out. Timeout was set to 300 seconds. Closing connection.
> 2015-09-25 18:50:06,026 [BitServer-3] INFO o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.88.135:51675 <--> /10.10.88.135:31011.
> 2015-09-25 18:50:06,032 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.88.135:31010 <--> /10.10.88.133:51612 (user client). Closing connection.
> java.lang.OutOfMemoryError: Java heap space
> {code}
> drillbit.out
> {code}
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main-SendThread(atsqa4-133.qa.lab:5181)"
> Exception in thread "WorkManager.StatusThread" java.lang.OutOfMemoryError: Java heap space
> 2015-09-25 18:53:52
> Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode):
> {code}
> jstack
> {code}
> [Fri Sep 25 18:53:29 ] # jstack 63205
> 63205: Unable to open socket file: target process not responding or HotSpot VM not loaded
> The -F option can be used when the target process is not responding
> {code}
> jstack -F
> {code}
> Attaching to process ID 63205, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 24.65-b04
> java.lang.RuntimeException: Unable to deduce type of thread from address 0x0000000004093800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
> at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
> at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000004093800
> at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> ... 15 more
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
> at sun.tools.jstack.JStack.main(JStack.java:102)
> Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x0000000004093800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
> at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
> at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> ... 6 more
> Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000004093800
> at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
> at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
> at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
> ... 12 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)