You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Rahul Challapalli (JIRA)" <ji...@apache.org> on 2015/06/18 00:39:01 UTC
[jira] [Created] (DRILL-3306) Concurrently Running hive queries
results in a deadlock situation
Rahul Challapalli created DRILL-3306:
----------------------------------------
Summary: Concurrently Running hive queries results in a deadlock situation
Key: DRILL-3306
URL: https://issues.apache.org/jira/browse/DRILL-3306
Project: Apache Drill
Issue Type: Bug
Components: Storage - Hive
Reporter: Rahul Challapalli
Assignee: Venki Korukanti
Priority: Critical
I tried submitting ~20 queries using 10 concurrent threads. Now the drillbit goes into an unresponsive state. Now even trying to launch a new sqlline session hangs (until hive.metastore.client.socket.timeout expires)
Below is the jstack output
{code}
Attaching to process ID 4662, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
Deadlock Detection:
java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fc0f8630000
at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
... 15 more
Can't print deadlocks:Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fc0f8630000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
... 6 more
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fc0f8630000
at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
... 12 more
{code}
Jstack Output of the hive metastore
{code}
Attaching to process ID 32320, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.65-b04
Deadlock Detection:
java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f1abd191800
at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
... 15 more
Can't print deadlocks:Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f1abd191800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162)
at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
... 6 more
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f1abd191800
at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158)
... 12 more
{code}
Observing from the web ui, after every 10 mins one query changes state from pending to running. All the queries in the running state did not complete and I stopped the drillbit after waiting for 30 mins
Also at this stage attempting to stop the drillbit does not succeed and the script internally call the kill command on the process
I attached the contents of drillbit.log file. Let me know if you need anything else
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)