You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Khurram Faraaz (JIRA)" <ji...@apache.org> on 2016/02/01 12:56:39 UTC
[jira] [Created] (DRILL-4330) Long running SQL query hangs once
Foreman node is killed
Khurram Faraaz created DRILL-4330:
-------------------------------------
Summary: Long running SQL query hangs once Foreman node is killed
Key: DRILL-4330
URL: https://issues.apache.org/jira/browse/DRILL-4330
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.4.0
Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz
Summary : Once Foreman node Drillbit is killed, long running query just hangs and no profile information is written to Web UI. That long running query was issued from the Foreman node.
MapR Drill 1.4.0 GA
MapR FS 5.0.0 GA
JDK8
4 node CentOS cluster
./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
Issue a long running select query over JSON data
Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill -9 PID
The long running query hangs on sqlline prompt, there are no messages/errors/Exceptions reported on sqlline prompt.
On the Web UI there is no profile reported for the long running query that was running on the Drillbit that was killed.
Question (1) : Why was there no profile reported/written on the Web UI for that long running query ? In a real production scenario user will not know what query was under execution at the point when Foreman went down.
Question (2) : Why does the long running query not terminate, once the foreman was killed ? from the drillbit.log snippet we do not see any CANCELED/TERMINATED message for that query, why ?
Snippet from drillbit.log on the foreman node.
{noformat}
2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 1 ms
2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 90ms total, 90.891938ms avg, 90ms max.
2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 18.280000 μs, Latest start: 18.280000 μs, Average start: 18.280000 μs .
2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
{noformat}
Doing kill -3 PID on the non foreman node for the Drillbit process gives us stack trace in drillbit.out
{noformat}
2016-02-01 11:03:31
Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
"qtp801808302-129" #129 prio=5 os_prio=0 tid=0x00007f7ad8127000 nid=0xaad waiting on condition [0x00007f7ab62fb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
at java.lang.Thread.run(Thread.java:745)
"DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000007b1283570> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-80-acceptor-0-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #80 prio=5 os_prio=0 tid=0x00007f7ad86df000 nid=0x7a7c runnable [0x00007f7ab64fd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000007b1283570> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-79-selector-ServerConnectorManager@6591a1ee/23" #79 prio=5 os_prio=0 tid=0x00007f7ad1368800 nid=0x7a7b runnable [0x00007f7ab65fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13f70d0> (a sun.nio.ch.Util$2)
- locked <0x00000007b13f70c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13f6fa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-78-selector-ServerConnectorManager@6591a1ee/22" #78 prio=5 os_prio=0 tid=0x00000000026c2800 nid=0x7a7a runnable [0x00007f7ab66ff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13f62e0> (a sun.nio.ch.Util$2)
- locked <0x00000007b13f62d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13f61b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-77-selector-ServerConnectorManager@6591a1ee/21" #77 prio=5 os_prio=0 tid=0x00007f7b233e5000 nid=0x7a79 runnable [0x00007f7ab6800000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13f54f0> (a sun.nio.ch.Util$2)
- locked <0x00000007b13f54e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13f53c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-76-selector-ServerConnectorManager@6591a1ee/20" #76 prio=5 os_prio=0 tid=0x00007f7ad86dd000 nid=0x7a78 runnable [0x00007f7ab6901000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13f4700> (a sun.nio.ch.Util$2)
- locked <0x00000007b13f46f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13f45d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-74-selector-ServerConnectorManager@6591a1ee/19" #74 prio=5 os_prio=0 tid=0x00007f7ad13cb800 nid=0x7a77 runnable [0x00007f7ab6a01000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13f3910> (a sun.nio.ch.Util$2)
- locked <0x00000007b13f3900> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13f37e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-73-selector-ServerConnectorManager@6591a1ee/18" #73 prio=5 os_prio=0 tid=0x00000000026c1000 nid=0x7a76 runnable [0x00007f7ab6b03000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13f2b20> (a sun.nio.ch.Util$2)
- locked <0x00000007b13f2b10> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13f29f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-72-selector-ServerConnectorManager@6591a1ee/17" #72 prio=5 os_prio=0 tid=0x00007f7b233e3800 nid=0x7a75 runnable [0x00007f7ab6c04000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13f1ce0> (a sun.nio.ch.Util$2)
- locked <0x00000007b13f1cd0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13f1bb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-71-selector-ServerConnectorManager@6591a1ee/16" #71 prio=5 os_prio=0 tid=0x00007f7ad86db000 nid=0x7a74 runnable [0x00007f7ab6d05000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b13e14e0> (a sun.nio.ch.Util$2)
- locked <0x00000007b13e14d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b13e13b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
...
"Scheduler-897230287" #62 prio=5 os_prio=0 tid=0x00007f7b23302800 nid=0x7a50 waiting on condition [0x00007f7ab770d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007bb901728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"qtp801808302-61-selector-ServerConnectorManager@6591a1ee/7" #61 prio=5 os_prio=0 tid=0x00007f7b233b8000 nid=0x7a4f runnable [0x00007f7ab780e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b12fb2a0> (a sun.nio.ch.Util$2)
- locked <0x00000007b12fb290> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b12fb178> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
...
"Curator-PathChildrenCache-1" #52 daemon prio=5 os_prio=0 tid=0x00007f7b2308c000 nid=0x7a47 waiting on condition [0x00007f7ab9179000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007bde07d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"WorkManager.StatusThread" #40 daemon prio=5 os_prio=0 tid=0x00007f7b229af000 nid=0x7a45 waiting on condition [0x00007f7ab825a000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.drill.exec.work.WorkManager$StatusThread.run(WorkManager.java:328)
"Curator-PathChildrenCache-0" #51 daemon prio=5 os_prio=0 tid=0x00007f7b2170f800 nid=0x79b2 waiting on condition [0x00007f7ab927a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b6b98818> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"BitServer-2" #21 daemon prio=10 os_prio=0 tid=0x00007f7b219d3800 nid=0x7981 runnable [0x00007f7ab940d000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:148)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
"UserServer-1" #41 daemon prio=10 os_prio=0 tid=0x00007f7b21846800 nid=0x797f runnable [0x00007f7abb1a6000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:148)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
"Curator-ServiceCache-0" #49 daemon prio=5 os_prio=0 tid=0x00007f7b2184d800 nid=0x796e waiting on condition [0x00007f7abc95d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b68066c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Curator-Framework-0" #45 daemon prio=5 os_prio=0 tid=0x00007f7b21064800 nid=0x77ef waiting on condition [0x00007f7aba9a6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b6806b90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"main-EventThread" #44 daemon prio=5 os_prio=0 tid=0x00007f7b21043000 nid=0x77ee waiting on condition [0x00007f7abaaa7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b6806e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"main-SendThread(centos-01.qa.lab:5181)" #43 daemon prio=5 os_prio=0 tid=0x00007f7b20ec9000 nid=0x77ed runnable [0x00007f7abb02d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007b5b00220> (a sun.nio.ch.Util$2)
- locked <0x00000007b5b00230> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b5b001d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070)
"Curator-ConnectionStateManager-0" #42 daemon prio=5 os_prio=0 tid=0x00007f7b2223d000 nid=0x77ec waiting on condition [0x00007f7ababa8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b6807088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
...
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f7b20cf8000 nid=0x7529 in Object.wait() [0x00007f7ac46d5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000007b6807f58> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7b20cf0000 nid=0x7528 in Object.wait() [0x00007f7ac47d6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000007b5b00d88> (a java.lang.ref.Reference$Lock)
...
Heap
garbage-first heap total 4194304K, used 293887K [0x00000006c0000000, 0x00000006c0204000, 0x00000007c0000000)
region size 2048K, 145 young (296960K), 21 survivors (43008K)
Metaspace used 43409K, capacity 43751K, committed 44284K, reserved 1087488K
class space used 5383K, capacity 5510K, committed 5628K, reserved 1048576K
2016-02-01 11:04:08
Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
"qtp801808302-130" #130 prio=5 os_prio=0 tid=0x00007f7ad8125000 nid=0xc5b waiting on condition [0x00007f7ab61fa000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
at java.lang.Thread.run(Thread.java:745)
"DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000007b1283570> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)